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Preface 



1 Introduction 

Imagine a virtual world with digital creatures that looks like real life, sounds like real 
life, and even feels like real life. Imagine a virtual world not only with nice three- 
dimensional graphics and animations, but also with realistic physical laws and forces. 
This virtual world could be familiar, reproducing some parts of our reality, or unfa- 
miliar, with strange “physical” laws and artificial life forms. 

As a researcher interested in the sciences of complexity, the idea of a conference 
about virtual worlds emerged from frustration. In the last few years, there has been an 
increasing interest in the design of artificial environments using image synthesis and 
virtual reality. The emergence of industry standards such as VRML [1] is an illustra- 
tion of this growing interest. At the same time, the field of Artificial Life has ad- 
dressed and modeled complex phenomena such as self-organization, reproduction, 
development, and evolution of artificial life-like systems [2]. One of the most popular 
works in this field has been Tierra designed by Tom Ray: an environment producing 
synthetic organisms based on a computer metaphor of organic life in which CPU time 
is the “energy” resource and memory is the “material” resource [3]. Memory is or- 
ganized into informational patterns that exploit CPU time for self-replication. Muta- 
tion generates new forms, and evolution proceeds by natural selection as different 
creatures compete for CPU time and memory space. 

However, very few works have used an Artificial Life approach together with 
Virtual Reality, or at least with advanced three-dimensional graphics. Karl Sims was 
probably one of the first researchers working in this direction. He designed a flexible 
genetic system to specify solutions to the problem of being a “creature” built of col- 
lections of blocks, linked by flexible joints powered by “muscles” controlled by cir- 
cuits based on an evolvable network of functions [4]. Sims embedded these “block 
creatures” in simulations of real physics, such as in water or on a surface. These ex- 
periments produced a bewildering and fascinating array of creatures, like the swim- 
ming “snake” or the walking ’’crab”. Demetri Terzopoulos and his colleagues have 
also created a virtual marine world inhabited by realistic artificial fishes [5]. They 
have emulated not only the appearance, movement, and behavior of individual ani- 
mals, but also the complex group behaviors evident in many aquatic ecosystems. Each 
animal was modeled holistically as an autonomous agent situated in a simulated 
physical world. 

Considering recent advances in both Artificial Life and Virtual Reality, catalyzed 
by the development of Internet , a unified approach seemed to be one of the most 
promising trend for the synthesis of realistic and imaginary virtual worlds. Thus, the 
primary goal of this conference was to set up an opportunity for researchers from both 
fields to meet and exchange ideas. 

In July 1998, the first international conference on virtual worlds was held at the 
International Institute of Multimedia. It brought together scientists involved in Virtual 
Reality, Artificial Life, Multi-agent Systems, and other fields of computer science and 
electronic art, all of whom share a common interest in the synthesis of digital worlds 
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on computers. The diversity and quality of the work reported herein reflects the im- 
pact that this new trend of research has had on the scientific community. 



2 The Proceedings 

The production of these proceedings was a major task, involving all the authors and 
reviewers. As the editor, I have managed the proceedings in a classical way. Every 
contribution that was accepted for presentation at the conference is in the proceedings. 
The program committee felt that these papers represented mature work of a level 
suitable for being recorded, most of them without modification, some of them requir- 
ing modifications to be definitively accepted. Besides the classical goal of a proceed- 
ings volume, the idea was to recapture in print the stimulating mix of ideas and works 
that were presented. Therefore, the papers are organized to reflect their presentation at 
the conference. There were three invited plenary speakers: Nadia Thalmann, Jeffrey 
Ventrella, and Yaneer Bar-Yam. Two of them choose to provide a printed version of 
their lecture. There were nine sessions, most of them in parallel, for a total of 36 pa- 
pers in all, recorded here roughly in the order in which they were presented. 

The material covered in these proceedings is diverse and falls naturally into a 
number of categories: Virtual Reality, Artificial Life, Multi-agent Systems, Complex- 
ity, Applications of Virtual Worlds, and last but not least. Virtual Worlds and Art. 
This collection of papers constitutes a good sample of works that appear necessary if 
we want to design large and realistic virtual worlds on the Internet in the near future. 
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4 Conclusion 

The terms “virtual worlds” generally refer to Virtual Reality applications or experi- 
ences. In this volume, I have extended the use of these terms to describe experiments 
that deal with the idea of synthesizing digital worlds on computers. Thus, Virtual 
Worlds (VW) could be defined as the study of computer programs that implement 
digital worlds with their own “physical” and “biological” laws. Constructing such 
complex artificial worlds seems to be extremely difficult to do in any sort of complete 
and realistic manner. Such a new discipline must benefit from a large amount of work 
in various fields: Virtual Reality, Artificial life. Cellular Automata, Evolutionary 
Computation, Simulation of Physical Systems, and more. Whereas Virtual Reality has 
largely concerned itself with the design of three-dimensional graphical spaces and 
Artificial Life with the simulation of living organisms, VW is concerned with the 
simulation of worlds and the synthesis of digital universes. 

This approach is something broader and more fundamental. Throughout the natural 
world, at any scale, from particles to galaxies, one can observe phenomena of great 
complexity. Research done in traditional sciences such as biology and physics has 
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shown that the basic components of complex systems are quite simple. It is now a 
crucial problem to elucidate the universal principles by which large numbers of sim- 
ple components, acting together, can self-organize and produce the complexity ob- 
served in our universe [6]. Therefore, VW is also concerned with the formal basis of 
synthetic universes. In this framework, the synthesis of virtual worlds offers a new 
approach for studying complexity. 

I hope that the reader will find in this volume many motivating and enlightening 
ideas. My wish is that this book will contribute to the development and further aware- 
ness of the new and fascinating field of Virtual Worlds. As of now, the future looks 
bright. 
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Abstract. This paper describes an efficient method to make an individual face 
for animation from several possible inputs and how to use this result for a 
realistic talking head communication in a virtual world. We present a method to 
reconstruct 3D facial model from two orthogonal pictures taken from front and 
side views. The method is based on extracting features from a face in a 
semiautomatic way and deforming a generic model. Texture mapping based on 
cylindrical projection is employed using a composed image from the two 
images. A reconstructed head is animated immediately and is able to talk with 
given text, which is transformed to corresponding phonemes and visemes. We 
also propose a system for individualized face-to-face communication through 
network using MPEG4. 

Keywords: Cloning, orthogonal pictures, DEED, texture mapping, talking head, 
visemes, animation 



1. Introduction 

Individualized facial communication is becoming more important in modern 
computer-user interfaces. To visualize ones own face in a virtual world and let people 
talk with given input, such as text or video, is now a very attractive research area. 
With the fast pace in computing, graphics and networking technologies, real-time 
face-to-face communication in a virtual world is now realizable. It is necessary to 
reconstruct an individual head in an efficient way to decrease data transmission size, 
and send few parameters for real-time performance. 

Cloning a real person's face has practical limitations in the sense of time, simple 
equipment and realistic shape. We present our approach to clone a real face from two 
orthogonal views, emphasizing accessibility for anybody with low price equipment. 
Our method to give animation structure on a range data from a laser scanner or 
stereoscopic camera is also described, but not in detail. This is because of the high 
price of this equipments. Therefore, it may not be a very practical idea to make use of 
them. The main idea is to detect feature points and modify a generic model with 
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animation structure. After creating virtual clones, we can use them to animate and 
talk in a virtual world. 

The organization of this paper is as follows. We give a review in Section 2 with 
classification for existing methods to get a realistic face reconstruction and talking 
head. In Section 3, we describe the idea of a system for individualized face-to-face 
communication through a network and our system for creating/animating talking head 
with given text. Section 4 is dedicated to the reconstruction process from feature 
detection to texture mapping and then the detailed process for talking head is 
explained in Section 5. Finally conclusion is given. 



2. Related Work 

2.1. Face Cloning for Animation 

There have been many approaches to reconstruct a realistic face in a virtual world. 
There are many possible ways such as using a plaster model [12][1], or interactive 
deformation and texture mapping [2] [15], which are time-consuming jobs. More 
efficient methods are classified into four categories. 



Laser Scanning In range image vision system some sensors, such as laser scanners, 
yield range images. For each pixel of the image, the range to the visible surface of the 
objects in the scene is known. Therefore, spatial location is determined for a large 
number of points on this surface. An example of commercial 3D digitizer based on 
laser-light scanning is Cyberware Color Digitizer™ [14]. 



Stripe Generator As an example of structured light camera range digitizer, a light 
striper with a camera and stripe pattern generator can be used for face reconstruction 
with relatively cheap equipment compared to laser scanners. With information of 
positions of projector and camera and stripe pattern, a 3D shape can be calculated. 
Proesmans et al. [13] shows a good dynamic 3D shape using a slide projector, by a 
frame-by-frame reconstruction of a video. However, it is a passive animation and new 
expressions cannot be generated. 



Stereoscopy A distance measurement method such as stereoscopy can establish the 
correspondence at certain characteristic points. The method uses the geometric 
relation over stereo images to recover the surface depth. C3D 2020 capture system [3] 
by the Turing Institute produces many VRML models using stereoscopy method. 

Most of the above methods concentrate on recovering a good shape, but the 
biggest drawback is that they provide only the shape without structured information. 
To get a structured shape for animation, the most typical way is to modify an 
available generic model with structural information such that eyes, lips, nose, hair and 
so on. Starting with a structured facial mesh, Lee et al. [13] developed algorithms that 
automatically construct functional models of the heads of human subjects from laser- 
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scanned range and reflection data [14]. However, the approach based on 3D 
digitization to get range data often requires special purpose high-cost hardware. 
Therefore, a common way of creating 3D objects is the reconstruction from 2D photo 
information, which is accessible at a low price. 



Modification with Feature Points on Pictures There are faster approaches to 
reconstruct a face shape from only a few pictures of a face. In this method, a generic 
model with an animation structure in 3D is provided in advance and a limited number 
of feature points. These feature points are the most characteristic points to recognize 
people, detected either automatically or interactively on two or more orthogonal 
pictures, and the other points on the generic model are modified by a special function. 
Then 3D points are calculated by just combining several 2D coordinates. Kurihara 
and Arai [9], Akimoto et al. [4], Ip and Yin [7], and Lee et al. [16] use an interactive, 
semiautomatic or automatic methods to detect feature points and modify a generic 
model. Some have drawbacks such as too few points to guarantee appropriate shape 
from a very different generic head or accurate texture fitting, or automatic methods, 
which are not robust enough for satisfactory result, like simple filtering and texture 
image generation using simple linear interpolation blending. 



2.2. Talking Head 

There are two approaches for the synthesis of talking heads. Pearce et al. [17] have 
used an approach to create an animation sequence with the input being a string of 
phonemes corresponding to the speech. In this case, the face is represented by a 3D 
model and animated by altering the position of various points in the 3D model. A the 
more recent work of Cohen and Massaro [18], english text is used as the input to 
generate the animation. The alternative is the image-based morphing approach. Ezzat 
and Poggio [19] have proposed a method of concatenating a collection of images, 
using a set of optical flow vectors to define the morphing transition paths, to create an 
animated talking head. In another recent work of Cosatto and Graf [20], they have 
proposed an automatic method of extracting samples from a video sequence of a 
talking person using image recognition techniques. In this case, the face image is 
being partitioned into several facial parts and later combined to generate a talking 
head. They focused to reduce the number of samples that are needed and photo- 
realistic movements of lips. 



2.3. Shared Virtual World 

There have been numerous works being done on the topic of shared virtual 
environments. In most of these works [23][24][25], each user is represented by a 
fairly simple embodiment, ranging from cube-like appearances, non-articulated 
human-like or cartoon-like avatars to articulated body representations using rigid 
body segments. In the work of Pandzic et al. [26], a fully articulated body with skin 
deformations and facial animation is used. However, none of these works discussed 
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about the process of creating individual face model in an efficient manner. In other 
words, each individual face model usually takes hours of effort to complete. The 
MPEG-4 framework [30] under standardization can be used to achieve a low 
bandwidth face-to-face communication between two or more users, using 
individualized faces. 



3. System Overview 

In networked collaborative virtual environments, each user can be represented by a 
virtual human with the ability to feel like “being together” by watching each other’s 
face. Our idea for face-to-face communication through a network is to clone a face 
with texture as starting process and send the parameters for the reconstruction of the 
head model to other users in the virtual environment. After this, only animation and 
audio parameters are sent to others in real time to communicate. The actual facial 
animation can be done on a local host and all the users can see every virtual human’s 
animation. This provides a low bandwidth solution for having a teleconference 
between distanced users, compared to traditional video conferencing, which sends 
video stream in real time through network. At the same time, it is able to retain a high 
level of realism with individualized textured 3D head. 

The general idea of a system for individualized face-to-face communication 
through network is shown in Fig. 1. There are two hosts, hostl and host2. Each host 
has a generic model and a program to clone a person from a given input such as two 
orthogonal pictures or range data. A person in hostl is reconstructed with pictures or 
range data obtained in any range data equipment. The parameters for reconstruction 
will be sent to host2 through network. They are texture image data, texture 
coordinates and either modification parameters or geometric position data for points 
on a 3D head surface. The same procedure happens in host2. Finally, both hostl and 
host2 have two 3D heads which are textured. Although we send a texture image, 
which takes large bandwidth, it is only one time process for reconstruction. After this 
process, only animation parameters and audio data are sent to another host through 
network. With animation parameters given, each host will animate two heads in their 
own platforms. 

The part of this system showing the proceedure of producing an individualized 
talking head is shown in Fig. 2. We reconstruct a real face from a given input. For 
reconstruction, only some points (so called feature points) are extracted from front 
and side views or only from front view if range data is available and then a generic 
model is modified. After reconstructing a head, it is ready to animate with given 
animation parameters. We use this model and apply it to talking head, which has 
animation abilities given text input. The input speech text is transformed to 
corresponding phonemes and animation parameters, so called visemes. In addition, 
expression parameters are added to produce final face with audio output. 
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Host 2 



Fig. 1. A system overview for individualized face-to-face communication through network. 



4. Face Cloning 

We reconstruct a face from two kinds of input, such as an orthogonal picture or range 
data in VRML format. The main idea is to modify a generic animation model with 
detected feature points and apply automatic texture mapping. In this paper, we focus 
on orthogonal picture input rather than range data input since the approach based on 
3D digitization to get a range data often requires special purpose (and high-cost) 
hardware and the process is quite similar to orthogonal picture case. 
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We consider hair outline, face outline and some interior points such as eyes, nose, 
lips, eyebrows and ears as feature points. 



Camera 



Laser Scanner 



Light stripper 



Stereoscopic 



Orthogonal Pictures 



Range Data 



Feature Detection 



J 




Generic Model 



^lodificatioi^^hap^^^J 



Input Speech Text 



Automatic 
Texture Fitting 



^Text-to-Speecl^yi^^ 






Phonemes 




JFacial^iiim^^ 



^Talkin^Iead^niinatioi^it| 



Fig. 2. Overall structure for individualized talking head. 
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4.1. Preparation and Normalization 

First, we prepare two 2D wire frames composed of feature points with predefined 
relations for front and side views. The frames are designed to be used as an initial 
position for the snake method later. Then we take pictures from front and side views 
of the head. The picture is taken with maximum resolution and the face is in the 
neutral expression and pose. 

To make the head heights of side and front views the same, we measure them, and 
choose one point from each view to matching them with corresponding points in 
prepared frame. Then we use transformation (scaling and translation) to bring the 
pictures to the wire frame coordinate, overlaying frames on pictures. 



4.2. Feature Detection 

We provide an automatic feature point extraction method with an interface for 
interactive correction when needed. There are methods to detect them just using 
special background information and predefined threshold [4] [7] and then use an edge 
detection method and apply threshold again. In addition, image segmentation by 
clustering method is used [4]. However, it is not very reliable since the boundary 
between hair and face and chin lines are not easy to detect in many cases. Moreover 
color thresholding is too sensitive and depends on each individual’s facial image and 
therefore requires many trials and experiments. We therefore use a structured snake, 
which has functionality to keep the structure of contours. It does not depend much on 
the background color and is more robust than simple thresholding method. 



Structured Snake First developed by Kass et al. [8] the active contour method, also 
called snakes, is widely used to fit a contour on a given image. Above the 
conventional snake, we add three more functions. First, we interactively move a few 
points to the corresponding position, and anchor them to keep the structure of points 
when snakes are involved, which is also useful to get more reliable result when the 
edge we would like to detect is not very strong. We then use color blending for a 
special area, so that it can be attracted by a special color [5]. When the color is not 
very helpful and Sobel operator is not enough to get good edge detection, we use a 
multiresolution technique [6] to obtain strong edges. It has two main operators, 
REDUCE with Gaussian operator and EXPAND. The subtraction produces an image 
resembling the result after Laplacian operators commonly used in the image 
processing. More times the REDUCE operator is applied stronger are the edges. 



4.3. Modifying a Generic Model 

3D Points from Two 2D Points We produce 3D points from two 2D points on 
frames with predefined relation between points from the front view and from the side 
view. Some points have x, y^, y,, z, so we take , y^ or average of y^ and for y 
coordinate (subscripts s and /mean side and front view). Some others have only x, y^ 




Won-Sook Lee et al. 



and others x, y^. Using predefined relation from a typical face, we get 3D position (x, 
y, z). 

Dirichlet Free-Form Deformations (DFFD) Distance-related functions have been 
employed by many researchers [4] [7] [9] to calculate the displacement of non-feature 
points related to feature points detected. We propose to use DFFD [11] since it has 
capacity for non-linear deformations as opposed to generally applied linear 
interpolation, which gives smooth result for the surface. We apply the DFFD on the 
points of the generic head. The displacement of non-feature points depends on the 
distance between control points. Since DFFD applies Voronoi and Delaunay 
triangulation, some points outside triangles of control points are not modified, the 
out-box of 27 points can be adjusted locally. Then the orginal shape of eyes and teeth 
are recovered since modifications may create unexpected deformation for them. Our 
system also provides a feedback modification of a head between feature detection and 
a resulted head. 




Generic Model 3D Feature lines Modified Head 

Fig. 3. A result of DFFD modification comparing with the original head. 



4.4. Automatic Texture Mapping 

To increase realism, we utilize texture mapping. Texture mapping needs a 2D texture 
image and coordinate for each point on a 3D head. Since our input is two pictures, 
texture image generation to combine them to one picture is needed. 



Texture Image Generation For smooth texture mapping, we assemble two images 
from the front and side views to be one. Boundaries of two pictures are detected using 
boundary color information or using detected feature points for face and hair 
boundaries. Since the hair shape is simple in a generic model, the boundary of a side 
view is modified automatically using information of back head profile feature points 
detected to have nice texture for back part of a neck. Then cylindrical projection of 
each image is processed. Two projected images are cropped at a certain position (we 
use eye extremes because eyes are important to keep at a high resolution), so that the 
range of combined image to make the final assembled image is 360°. Finally, a 
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multiresolution spline assembling method is used to produce one image for texture 
mapping preventing visible boundary for image mosaic. 




Cylindrical Projection of 
every points on a head 




BaryCentric 
Coordinates of every 
point on a head 
corresponding to 
Voronoi triangles of 
feature points 




Crop Lines 




Multiresolution technique 




Voronoi Triangulation of feature 
points on a head. 

6 Points added for a convex hull 
containing every point on a head 



Texture Coordinates of feature 
points and extra 6 points followed 
same projection and cropping lines 
with texture image 




Final Texture Coordinate of 
every point on a head. 
Eyes and Teeth Coordinates 
are decided earlier and used 
for same coordinates for 
every individual head. 



Fig. 4. Texture mapping process. 



Texture Fitting The main idea for the texture fitting is to map a 2D image on a 3D 
shape. Texture coordinates of feature points are calculated using detected position 
data and function applied for texture image generation. The problem for texture 
fitting is how to decide texture coordinates of all points on a surface of a head. We 
first apply a cylindrical projection of every point on a 3D head surface. Extra points 
are added to make a convex hull containing all points, so that a coordinate of every 
point is located on an image. Then the Voronoi triangulation on control (feature) 
points and extra points are processed and the local Barycentric coordinates of every 
point with a surrounding Voronoi triangle are calculated. Finally the texture 
coordinates of each point on a 2D-texture image are obtained using texture 
coordinates of control points and extra points and correspond Barycentric coordinate. 



4.5. Result 

A final textured head is shown in Fig. 5 with input images, whose process from 
normalization to texture mapping takes a few minutes. 







10 



Won-Sook Lee et al. 




Fig. 5. A final reconstructed head with two input images in left side. The back of head has 

proper texture too. 



5. Talking Head and Animation 

In this section, we will describe the steps for animating a 3D face model according to 
an input speech text, that has been reconstructed as described in the previous section. 

Firstly, the input text is being provided to a text-to-speech synthesis system. In this 
case, we are using the Festival Speech Synthesis System [21] that is being developed 
at the University of Edinburgh. It produces the audio stream that is subsequently 
played back in synchronization with the facial animation. The other output that is 
needed from this system is the temporized phoneme information, which is used for 
generating the facial animation. 

In the facial animation system [22] that is used in our work, the basic motion 
parameter for animating the face is called Minimum Perceptible Action (MPA). Each 
MPA describes a corresponding set of visible features such as movement of 
eyebrows, jaw, or mouth occurring as a result of muscle contractions and pulls. 

In order to produce facial animation based on the input speech text, we have 
defined a visual correlation to each phoneme, which are commonly known as 
visemes. In other words, each viseme is simply corresponding to a set of MPAs. In 
Fig. 6, it shows a cloned head model with a few visemes corresponding to the 
pronunciation of the words indicated. 

With the temporized phoneme information, facial animation is then generated by 
concatenating the visemes. We have limited the set of phonemes to those used in the 
Oxford English Dictionary. However, it is easy to extend the set of phonemes in order 
to generate facial animation for languages other than English, as this can be easily 
done by adding the corresponding visemes. 

The facial animation described so far is mainly concerned with the lip movement 
corresponding to the input speech text. However, this animation will appear artificial 
without the eyes blinking, and movement of the eyes and head. Therefore, we have 
also included random eyes blinking and movement of the eyes and head to add more 
realism into the generated animation. An analysis of the input speech text can also be 
done to infer the emotional state of the sentence. Then, the corresponding emotional 
facial expression can also be applied to the animation. Eor example, the eyebrows can 
be raised when the emotional state is surprise. 
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Fig. 6. A cloned head model with some examples of visemes for the indicated words. Our 
talking head system utilizes 44 visemes and some facial expression parameters. 



An example of cloned persons interacting in a shared virtual environment is shown 
in Fig. 7. Two kinds of generic bodies, female and male, are provided in advance. 
The generic bodies can be adjusted according to several ratios using Bodylib [28]. 
We connect individualized heads onto bodies by specifying a transformation matrix. 
Basically we need four types of data, namely the geometrical shape, animation 
structure, texture image and texture coordinates. In our case, every individualized 
head shares the same animation structure. Final rendering is then produced in real 
time. Our whole process from individualization to final talking head with body in a 
virtual world takes only few minutes. 




Fig. 7. Cloned persons interacting in shared virtual environments. 



6. Conclusion and Future Research 

In this paper, we proposed an efficient way of creating the individual face model, in 
terms of time needed for the creation and transmitting the information to other users, 
and sending parameters for facial animation. The input of two orthogonal pictures can 
be easily obtained from any kind of conventional camera. The process of 
individualization consists of several steps including feature detection on 2D pictures. 
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Dirichlet Free-Form Deformations for modification of a generic model and automatic 
texture mapping. The reconstructed heads connected to given bodies are used in a 
virtual environment. The result is then used immediately to produce an individualized 
talking head, which is created from given text and viseme database. This whole 
process can be achieved together with the MPEG-4 framework [27] to create a low 
bandwidth face-to-face communication between two or more users. 

In the television broadcasting industry, the reconstruction method that we have 
described in this paper will allow actors to be cloned rapidly. Then, the television 
producer will be able to direct the virtual actors to say the lines and have the 
appropriate facial expressions in a virtual studio. This will obviously reduce the cost 
of production. 
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Abstract. Rendering and animating in real-time a multitude of articulated 
characters presents a real challenge and few hardware systems are up to the 
task. Up to now little research has been conducted to tackle the issue of real- 
time rendering of numerous virtual humans. However, due to the growing in- 
terest in collaborative virtual environments the demand for numerous realistic 
avatars is becoming stronger. This paper presents a hardware-independent 
technique that improves the display rate of animated characters by acting on the 
sole geometric and rendering information. We first review the acceleration 
techniques traditionally in use in computer graphics and highlight their suit- 
ability to articulated characters. Then we show how impostors can be used to 
render virtual humans. Finally we introduce a concrete case study that demon- 
strates the effectiveness of our approach. 



1 Introduction 

Even though our visual system is not deceived yet when confronted with virtual hu- 
mans, our acceptance of virtual characters has greatly improved over the past few 
years. Today’s virtual humans faithfully embody real participants in collaborative 
virtual environments for example and are even capable of conveying emotions 
through facial animation [1]. Therefore, the demand for realistic real-time virtual 
humans is becoming stronger. Yet, despite the ever-increasing power of graphics 
workstations, rendering and animating virtual humans remain a very expensive task. 
Even a very high-end graphics system can have trouble sustaining a sufficient frame 
rate when it has to render numerous moving human figures commonly made up of 
thousands of polygons. While there is little douht that hardware systems will eventu- 
ally be fast enough, a few simple yet powerful software techniques can be used to 
speed up rendering of virtual humans by an order of magnitude. 

Because 3D chips were not affordable or did not even exist in the 80s, video game 
characters, human-like or not, were then represented with 2D sprites. A sprite can be 
thought of as a block of pixels and a mask. The pixels give the color information of 
the final 2 -h-D image while the mask corresponds to a binary transparency channel. 
Using sprites, a human figure could easily be integrated into the decor. As more com- 
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puling power was available in the 90s, the video game industry shifted towards 3D. 
However, the notion of sprites can also be used in the context of 3D rendering. This 
has been successfully demonstrated with billboards, which are basically 3D sprites, 
used for rendering very complex objects like trees or plants. In our opinion image- 
based rendering can also be used in the case of virtual humans by relying on the in- 
trinsic temporal coherence of the animation. 

Current graphics systems rarely take advantage of temporal coherence during ani- 
mation. Yet, changes from frame to frame in a static scene are typically very small, 
which can obviously be exploited [9]. This still holds true for moving objects such as 
virtual humans providing that the motion remains slow in comparison with the 
graphics frame rate. We present in this paper a software approach to accelerated ren- 
dering of moving, articulated characters, which could easily be extended to any 
moving and/or self-deforming object. Our method is based on impostors, a combina- 
tion of traditional level-of-detail techniques and image-based rendering and relies on 
the principle of temporal coherence. It does not require special hardware (except 
texture mapping and Z-buffering capabilities, which are commonplace on high-end 
workstations nowadays) though fast texture paging and frame buffer texturing is 
desirable for optimal performance. 

The next section gives an overview of the existing schemes that are used to yield 
significant rendering speedups. Section 3 briefly describes our virtual human model 
and discusses the limitations of geometry level-of-detail techniques. The notion of 
dynamically generated impostors is introduced in section 4 with the presentation of an 
algorithm that generates virtual humans from previous rasterized images. In section 5 
the duration of the validity of the image cache is discussed. The following section 
then shows how this method can be embedded in a large-scale simulation of a human 
flow. Finally section 7 concludes the paper with a summary of the results we obtained 
and leads on to some possible future work. 



2 Background 

There exist various techniques to speed up the rendering of a geometrical scene. They 
roughly fall into three categories: culling, geometric level-of-detail and image-based 
rendering which encompasses the concept of image caching. They all have in com- 
mon the idea of reducing the complexity of the scene while retaining its visual char- 
acteristics. Besides, they can often be combined to produce better results, as shown in 
sections 3 and 4. 

Culling algorithms basically discard objects or parts of objects that are not visible 
in the final rendered image: an object is not sent to the graphics pipeline if it lies 
outside the viewing frustum (visibility culling) or if it is occluded by other parts of 
the scene (occlusion culling). Luebke and Georges described an occlusion culling 
system well suited for highly occluded architectural models that determines poten- 
tially visible sets at render-time [3]. One major drawback of occlusion culling algo- 
rithms is that they usually require a specific organization of the whole geometry data- 
base: the scene is typically divided into smaller units or cells to accelerate the culling 
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process. Therefore, they perform poorly on individual objects. Because of this re- 
striction it is no use trying to apply occlusion techniques to virtual humans if they are 
considered individually as we do. Nevertheless, occlusion culling might be contem- 
plated at a higher level if the virtual humans are placed in a densely occluded world, 
as could be the case when simulating a human crowd for instance. 

Geometric level of detail (LOD) attempts to reduce the number of rendered poly- 
gons by using several representations of decreasing complexity of an object. At each 
frame the appropriate model or resolution is selected. Typically the selection criterion 
is the distance to the viewer although the object motion is also taken into account 
(motion LOD) in some cases. The major hindrance to using LOD is related to the 
problem of multi-resolution modeling, that is to say the automatic generation from a 
3D object of simpler, coarser 3D representations that bear as strong a resemblance as 
possible to the original object. Heckbert and Garland [4] presented a survey of mesh 
simplification techniques. More recently, some more work has been done to provide 
visually better approximations [5,6]. Apart from the problem of multi-resolution 
modeling, it is usually quite straightforward to use LODs in virtual reality applica- 
tions. Funkhouser and Sequin described a benefit heuristic [15] to select the best 
resolution for each rendered object in the scene. Several factors are taken into account 
in their heuristic: image-space size so as to diminish the contribution of distant ob- 
jects, distance to the line of sight because our visual system perceives with less accu- 
racy objects on the periphery, motion because fast moving objects tend to appear 
blurred in our eyes. In addition, objects can be assigned different priorities since they 
all may play not an equal role in the simulation. Finally, hysteresis is also taken into 
consideration in order to avoid the visually distracting effect that occurs when the 
object keeps switching between two LODs. Recently Pratt et al. [14] have applied 
geometric LODs to virtual humans in large-scale, networked virtual environments. 
They used heuristic to determine the viewing distances that should trigger a LOD 
switch. Each simulated human has four different resolutions ranging from about 500 
polygons down to only three polygons. However, their lowest resolutions are proba- 
bly too coarse to be actually used unless the virtual human is only a few pixels high, 
in which case another technique than polygon rendering may be considered. 

Finally, image caching is based on the principle of temporal coherence during 
animation. The basic idea is to re-use parts of the frame buffer content over several 
frames, thus avoiding rendering for each frame the whole scene from scratch [2]. The 
hardware architecture Talisman [7] exploits this temporal coherence of the frame 
buffer: independent objects are rendered at different rates into separate image layers 
which are composited together at video rate to create the displayed image. In addi- 
tion, the software host is allowed to maintain a priority list for the image layers, 
which means that objects in the foreground can be updated more frequently than 
remote objects in the background. Maciel et al. [8] presented a navigation system of 
large environments, in which they introduced the concept of impostors. An impostor 
is some extremely simple geometry, usually texture-mapped planes, which retains the 
visual characteristics of the original 3D object. In their navigation system they re- 
placed clusters of objects with textured planes (i.e. impostors), one of the first at- 
tempts at software image-based rendering. But they do not generate the textures on 
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the fly. Hence their system requires a pre-processing stage during which the scene is 
parsed: N textures - where N corresponds to the number of potential viewing direc- 
tions - are generated for every object and stored in memory. This approach is thereby 
likely to consume rapidly the texture memory. Shaufler and Stiirzlinger first intro- 
duced the notion of dynamically generated impostors [9]. They managed to solve the 
problem of texture memory usage since textures are generated on demand in their 
system. Since then two other software solutions using image-based rendering have 
been proposed concurrently and independently to accelerate walkthroughs of complex 
environments [10,11]. Lastly, Sillon et al. used impostors augmented with three- 
dimensional information for the visualization of urban scenery [18]. 

Although impostors appear to be one of the most promising techniques their use 
has mainly been restricted so far to walkthroughs of large environmenfs. We believe 
however that this technique can bring a substantial improvement to virtual humans 
too. 



3 A Multi-Resolution Virtual Human 

In this section we show how a multi-resolution virtual human can successfully be 
constructed and animated. We have designed a human modeler that solves the tradi- 
tional multi-resolution problem described above. By relying on implicit surfaces we 
avoid resorting to complicated algorithms and automatically generate multiple look- 
alike resolutions of a model. Finally, we explain why a mulli-resolution human does 
nol provide a sufficient gain in rendering time. 



3.1 Multi-Resolution Modeling 

Our real-time virtual human model consists of an invisible skeleton and a skin. The 
underlying skeleton is a hierarchy of joinfs thaf correspond to the real human main 
joints. Each joint has a set of degrees of freedom, rofation and/or franslation, which 
are constrained to authorized values based on real human mobility capabilities [12]. 
Unlike other attempts [14] we did not model a multi-resolution skeleton because our 
purpose was not to demonstrate the effectiveness of animation level-of-detail. Hand 
joinfs can be replaced with a single joint though. 

The skin of our virfual human is generafed by means of an in-house modeler fhaf 
relies on a multi-layer approach (Fig. 1): ellipsoidal metaballs and ellipsoids are em- 
ployed to represent the gross shape of bone, muscle and fat tissue [13]. Each primitive 
is attached to its proximal joint in the underlying human skeleton. The set of primi- 
tives then defines an implicif surface fhaf accurately approximates the real human 
skin. Sampling this implicit surface results in a polygonal mesh, which can be directly 
used for rendering. This polygonal mesh constitutes the skin of our virfual human 
model. 

Sampling fhe implicit surface is done as follows: we start by defining contours cir- 
cling around each limb link of the underlying skeleton. We then cast rays in a star- 
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shaped manner for each contour, with ray origins sitting on the skeleton link. For 
each ray, we compute the outermost intersection point with the implicit surface sur- 
rounding the link. The intersection is a sample point on the cross-section contour. 
Once all the sampling is done it is quite straightforward to construct a mesh from the 
sample points. Thus, we can generate more or less detailed polygonal meshes by 
simply varying the sample density i.e. the number of sampled contours as well as the 
number of sampled points per contour. And yet, since the implicit surface remains the 
same no matter the sampling frequency, the generated meshes look very similar (Fig. 
2 ). 




Fig. 1. Multi-layered model [131 it.- n j u cj ■ i ■. 

® ■' fig. 2. Body meshes ot decreasing complexity 



As for the head, hands and feet, we still have to rely on a traditional decimation tech- 
nique to simplify the original mesh. Manual intervention is still needed at the end of 
this process to smooth the transition between LODs. The body extremities can clev- 
erly be replaced with simple textured geometry for the lowest resolution (Fig. 3), 
which dramatically cuts down the number of triangles. 




Fig. 3. Levels of detail 
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3.2 Animation 

The skeleton of our virtual human comprises a total of 74 DOFs corresponding to the 
real human joints plus a few global mobility nodes, which are used to orient and po- 
sition the virtual human in the world. In the broad lines, animating a virtual human 
consists in updating this skeleton hierarchy including the global mobility joints at a 
fixed frame rate. There exist several techniques to feed the joints with new an- 
gle/position values. Motion capture for instance is used to record the body joint val- 
ues for a given lapse of time. The motion can later be played back on demand for any 
virtual human. Key frame animation is another popular technique in which the ani- 
mator explicitly specifies the kinematics by supplying key-frame values and lets the 
computer interpolate the values for the in-between frames. 

During animation the appropriate resolution is selected for each individual ac- 
cording to the euclidian distance to the viewpoint. Therefore far objects and those on 
the periphery contribute less to the final image. Note that we could also take the gen- 
eral motion of the virtual human into account. Finally at a higher level, typically the 
application layer, virtual humans could be assigned different rendering priorities too. 
For example, in the context of a human crowd simulation where individuals move 
and act in clusters the application could decide to privilege some groups. 



3.3 Limitations of Geometric LODs 

LOD is a very popular technique probably because of its simplicity. It is no wonder 
that LOD has widely been used in computer graphics, whether in virtual reality appli- 
cations or in video games. There are some limitations to LOD though. First, shading 
artifacts may arise when the geometry is extremely simplified. This is conspicuous 
when Gouraud shading is used because the shading function is evaluated for fewer 
points, as the geometry is decimated. Popping is another recurrent problem: however 
look-alike two successive LODs may be, the switch can sometimes be obvious to the 
viewer. The most widespread solution to this problem relies on a transition zone to 
smooth the switch. In this special zone both LODs images are blended using a trans- 
parency channel. However, this method presents the disadvantage of displaying twice 
the geometry for a given lapse of time. Lastly, geometric LOD has physical limits in 
the sense that it is not possible to represent properly a human being with fewer than a 
few hundred polygons. For example, if we consider textured bounding boxes as the 
lowest resolution, the number of triangles for our model still amounts to about two 
hundred in all. Such a representation allows a scene that is made up of dozens of 
virtual actors to be refreshed at a high frame rate - typically 20 Hz - but to the detri- 
ment of the visual quality that drops to a level that is barely acceptable. For all that, 
LOD remains a very efficient technique that can be used without difficulty in con- 
junction with impostors. 
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4 Animated Impostors for Articulated Characters 

As first defined by Maciel et al. [8] an impostor is in essence some very simple ge- 
ometry that manages to fool the viewer. As traditionally the case in the existing lit- 
erature, what we mean by impostor is a set of transparent polygons onto which we 
map a meaningful, opaque image. More specifically our impostor corresponding to a 
virtual human is a simple textured plane that rotates to face continuously the viewer. 
The image or texture that is mapped onto this plane is merely a “snapshot” of the 
virtual human. Under these conditions if we take for granted that the picture of the 
virtual human can be re-used over several frames then we have virtually decreased 
the polygon complexity of a human character to a single plane (Fig. 4). 




Fig. 4. A Brazilian football player and its impostor 



4.1 Impostor Refreshment Approach 

The texture that is mapped onto the transparent plane still needs to be refreshed from 
time to time because of the virtual human’s mobility or camera motion. Whenever the 
texture needs to be updated a snapshot of the virtual human is taken. This process is 
done in three steps: 

1 . Set up an off-screen buffer that will receive the snapshot 

2. Place the virtual human in front of the camera in the right posture 

3. Render the actor and copy the result into texture memory 

The first stage typically comes down to clearing the buffer. The buffer should be a 
part of the frame buffer so as to benefit from hardware acceleration. The purpose of 
the second step is to set up the proper view to take a picture: first, we let the actor 
strike the right pose depending on joint values. Second, it is moved in front of the 
camera or alternately, the camera itself is moved. Note that the latter is preferable 
because the skeleton hierarchy remains unchanged in this case, which may save some 
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time. In the last stage the virtual human is rendered and the resulting image is copied 
into texture memory. Once the texture has been generated there only remains to ren- 
der the textured billboard to have a virtual human on screen. The whole process is 
hardly any slower than rendering the actual 3D geometry of the virtual human for the 
following reasons: setting the off-screen buffer up can be done once for all in a pre- 
processing step. It chiefly consists in adjusting the viewing frustum to the virtual 
human. Furthermore, this buffer can be re-used to generate several textures of differ- 
ent actors providing that they have approximately the same size. Clearing the buffer is 
definitely not a costly operation. Letting the virtual human assume the right posture 
and rendering it would also have to be carried out if the real geometry was used in- 
stead. Finally, if the hardware features frame buffer texturing, the frame buffer to 
texture memory copy is performed within no time. If not, clearing the off-screen 
buffer and transferring its content still remain all the less costly that the buffer size is 
small, typically 128 by 128 pixels or less. As a consequence even in the worst cases 
(very high texture refreshment rates), impostors prove not to be slower than rendering 
the actual 3D geometry. 



4.2 Impostor Viewing and Projection 

For the sake of clarity, when speaking respectively in the following of texture win- 
dows and scene window, we will mean the off-screen buffers where textures of vir- 
tual humans are generated and the window where the final, global scene with impos- 
tors is rendered. 





Drawing a. represents a top view of a 
scene containing two articulated char- 
acters. Each character has its own coor- 
dinates system. The subjective camera 
coordinate system defines the user’s 
viewpoint and corresponds to the camera 
in the scene window. All coordinates 
systems are expressed with respect to the 
world. 



Drawing b. shows a top view of the 
same scene rendered with impostors. 
Each character’s geometry is replaced 
with a single textured plane. Each plane 
is oriented so that its normal vector 
points back to the eye i.e. the origin of 
the subjective camera coordinate system. 
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Drawing c. explains how the textures 
mapped onto the planes in drawing b. are 
generated i.e. how snapshots of virtual 
humans are taken. In the drawing there are 
two orthographic camera coordinates sys- 
tems corresponding to the cameras in the 
texture windows associated with each 
articulated character. The viewing direc- 
tion Vo of the orthographic camera is 
determined by the eye’s position and a 
fixed point (later referred to as Ta) of the 
articulated character, as will be explained. 
Drawing d. is a more detailed version of 
the previous drawing. It mainly shows how 
the orthographic camera coordinate system 
is constructed. Vo is the unit vector de- 
rived from Ta and the position of the sub- 
jective camera. The two other unit vectors 
that give the orthographic camera’s orien- 
tation Ro are determined as follows: the 
first vector Ho is the cross product of Vo 
and Us where Us is the normalized up 
vector of the subjective camera’s frame. 
The cross product of Ho and Vo then 
yields Uo. Note that this method ensures 
we obtain a correct frame (Ho^O) as long 
as the field of view is smaller than a half 
sphere. 



Finally, the camera of the texture window (termed orthographic camera so far) must 
be positioned so that the virtual human is always entirely visible and of a constant 
global size when we take a snapshot. We proceed in two steps to do so. First, an or- 
thographic projection is used because it incurs no distortion and actual sizes of objects 
are maintained when they are projected. Second, a fixed point named Ta is chosen in 
the body (it corresponds to the spine base) so that it is always projected onto the mid- 
dle of the texture window. Practically, Ta is determined to be the “middle” point of 
the virtual human assuming a standing posture in which its arms and hands are fully 
stretched above its head. Note that this posture is the one in which the virtual human’s 
apparent size is maximal. The orthographic camera is placed at distance D from Ta 
(D is a scalar). During the simulation, the camera is then moved at To = Ta - D.Vo. 
Hence Ta’s projection necessarily coincides with the center of the texture window. 
On account of the orthographic projection, D actually does not influence the virtual 
human’s projection size. On the other hand the viewing frustum does. Consequently, 
it must be carefully chosen so that no parts of the virtual character are culled away. 
This is done in a pre-processing stage in which the viewing frustum’s dimensions (in 
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the texture window) are set to the maximal size of the virtual human (i.e. with arms 
raised above its head). 



4.3 Several Texture Resolutions 

All the work has been realized on Silicon Graphics workstations using the Performer 
graphics toolkit. Because of hardware/software restrictions concerning textures, tex- 
ture windows have dimensions that are powers of two. Several texture windows of 
decreasing size can be associated with every virtual human. As the virtual human 
moves farther, which leads to smaller image-space size in the scene window, a 
smaller texture window can be used thus reducing texture memory consumption. 
Practically, we allocated 256x256, 128x128, 64x64 and 32x32 pixel texture windows 
for a simulation running at a 1280x1024 resolution. The largest windows were only 
used for close-up views. As always, it can be a bit difficult to strike the right balance 
between texture memory consumption and visual realism. In addition to better man- 
aging texture memory, using several texture windows of decreasing size also helps to 
reduce visual artifacts. As a matter of fact large textures of virtual humans that are 
mapped onto very small (far) planes might shimmer and flash as the impostors move. 
When several texture resolutions are employed, these artifacts tend to disappear be- 
cause the appropriate texture resolution is selected according to the impostor image- 
space size (in pixels). In a certain way this mimics the well-known effect obtained 
with “mip mapping”. Finally, when a texture is generated the appropriate LOD of the 
geometry can be chosen based on the texture window size. 



4.4 Main Current Limitation 

Replacing a polygonal model of a virtual human with a single textured plane may 
introduce visibility problems: depth values of the texels are unlikely to match those of 
the actual geometry, which may lead to incorrect visibility, as illustrated in figure 5. 
We did not address this issue in this paper. 



iimmetrv i/npostarr 




Fig. 5. Incorrect visibility 
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5 Image Cache Duration 

For static objects camera motion is the one and only factor to take into consideration 
for cache invalidation [9,11]. The algorithm that decides whether a snapshot of a 
virtual human is stale or not is obviously a bit more complex. However, that algo- 
rithm has to execute very quickly because it must be performed for every virtual 
human at each frame. In our approach we distinguish two main factors: self- 
deformation of the virtual actor and its global motion with respect to the camera. 



5.1. Virtual Humans as Self-Deforming Objects 

Virtual humans can be considered as self-deforming objects in the sense they do not 
keep a static shape, i.e. they can take different postures. The basic idea for a cache 
invalidation algorithm is that the viewer need not see every posture of the virtual 
humans to fully understand what action they are performing. A few key postures are 
often meaningful enough. We propose a simple algorithm that reflects the idea of sub- 
sampling the motion. The idea is to test distance variations between some pre-selected 
points in the skeleton. Using this scheme the virtual human is re-rendered if and only 
if the posture has changed significantly. 

Once we have updated the skeleton hierarchy we 
have direct access to joints’ positions in world 
space. It is therefore quite straightforward to com- 
pute distances (or rather squared distances to avoid 
unnecessary square roots) between some particular 
joints. In concrete terms we compare four distances 
(Fig. 6) with those stored when the texture was last 
generated. As soon as the variation exceeds a cer- 
tain threshold the texture is to be re-generated. Of 
course the thresholds depend on the precision the 
viewer demands. Furthermore, length variations are 
weighted with the distance to the viewer to decrease 
the texture refreshment rate as the virtual human 
moves away from the viewpoint. 

We found out that four tests suffice to reflect any significant change in the virtual 
human’s posture. Nevertheless, other tests should be performed if the simulation is 
meant to underscore some peculiar actions e.g. grasping of an object requires addi- 
tional testing of the hand motion. Similarly, it might be necessary to increase or de- 
crease the number of tests when using non-human characters. As a rule the number of 
limbs gives the number of tests to make. 



/■i 



Fig. 6. Posture variations to 
test 
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5.2. Relative Motion of the Actor in the Camera Frame 

Instead of testing independently camera motion and actor’s orientation we have come 
up with a simple algorithm that checks both in a single test. The algorithm’s main 
idea stems from the fact that every virtual human is always seen under a certain 
viewing angle which varies during simulation whether because of camera motion or 
actor’s motion. Yet, there is basically no need to know what factor actually caused the 
variation. 

In practice we test the variation of a “view” matrix, which corresponds to the trans- 
formation under which the viewer sees the virtual human. This matrix is plainly the 
product of the subjective camera matrix (camera in the scene window) and that of the 
articulated character. The cache invalidation algorithm then runs in pseudo-code as 
follows: 

For every virtual human at frame 0 /* Initialization stage */ 

Generate texture 
Store View Matrix 
End for 

For every virtual human at frame N>0 /* Simulation */ 

Compute new View Matrix 

Compute View Variation Matrix M (from previously stored 

view matrix to the newly computed one) . 
Compute the amplitude of the corresponding axis /angle 
If angle>threshold 
Generate texture 
Store current View Matrix 
End if 
End for 

Building the axis-angle representation of a rotation matrix M consists in finding its 
equivalent rotation axis as well as the angle to rotate about it [16]. Finally, the thresh- 
old that indicates when the texture is to be regenerated can be weighted once again 
with the euclidian distance from the impostor to the viewer. 



6. The Human Flow Test Bed 

Our work on impostors originates from earlier research on human crowd simulation 
that was carried out in our laboratory. However, simulating a human crowd intro- 
duces many parameters that alter the frame rate results. We preferred to use a simpler 
environment in order to assess reliably the gain of impostors on geometry. In our 
simulation twenty walking virtual humans keep circling. They all move at different 
yet constant speeds, along more or less long circles (Fig. 7). A fast walking engine 
handles the motion of the actors, collision between characters are not detected and 
finally every articulated character always lies in the field of vision so that visibility 
culling does not play a role. 
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The first graph shows the influence of the posture variation thresholds used in the 
texture cache invalidation mechanism. The other factor, that is the viewing angle, was 
deactivated throughout this test. Along the horizontal axis are noted the posture 
variation thresholds beyond which a texture is re-generated while the vertical axis 
shows the amount of time required for rendering the complete scene. The vertical axis 
is normalized with respect to the amount of time needed to render the whole scene 
using the real geometry. When the threshold is set to zero every change in the posture 
triggers a re-computation of the texture, in which case the rendering time logically 
exceeds the reference rendering time. Note that there is only a marginal difference of 
15% though, which clearly shows that impostors are hardly slower than the actual 
geometry even in the worst cases. Rendering time plummets when the variation 
threshold is increased: a threshold of 40%, 60% and 80% cuts respectively by two, 
three and five the rendering time. In practice there is no noticeable difference in the 
animation of the actors as long as the threshold is smaller than 15%. However, it 
makes perfect sense to set the threshold to a much higher limit (between 40% and 
80%) because the motion remains absolutely understandable. On the other hand it 
becomes hard to grasp that the actors are walking beyond 90%. 




Fig. 7. Simulating two football teams (left: impostors, right: real geometry) 





Graph 1.: Rendering speedup as a function of Graph 2.: Rendering speedup as a function of 
posture variation threshold the viewing angle variation threshold 
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The second test focuses on the impact of the viewing angle on the rendering time. 
Like previously, the vertical axis is normalized with respect to a reference rendering 
time (that needed to render the whole scene with the actual geometry). On the hori- 
zontal axis are marked the viewing angle thresholds in degrees. Similarly to the first 
test we disabled the other factor in the cache invalidation mechanism. The critical 
threshold is reached for a viewing angle variation of two degrees only. Viewing angle 
thresholds of 5, 8 and 17 degrees cut respectively by two, three and five the rendering 
time. We consider that there is no real degradation of the animation up to 20 degrees. 
Refreshment of the actors’ texture becomes too obvious beyond 30 degrees. 



7. Conclusion 

This paper shows that image-based rendering can be applied successfully to virtual 
humans. In particular, we have explained how to generate a texture representing a 
virtual human and proposed an Image cache invalidation algorithm that works for any 
articulated character and executes reasonably fast. Two major issues, which were 
beyond the scope of this paper, could be addressed in future work: 



1. Our texture refreshment algorithm performs on an individual basis. From the ap- 
plication point of view, the image cache invalidation mechanism should also con- 
sider virtual characters as a whole. Above all this would help achieve a quasi- 
constant frame rate, which is often regarded in virtual reality applications as more 
important than a high peak frame rate. 



2. Because depth information is lost when complex 3D geometry is replaced with an 
impostor, especially with a plane, visibility may not be correctly resolved. Schau- 
fler recently showed how to correct the visibility by directly modifying the depth 
value for every texel [17]. However, faster techniques could be investigated in the 
specific case of virtual humans. For example, the concrete problem depicted in 
figure 5 could be solved by decomposing the impostor’s plane into several planes 
e.g. one for each major body part. 
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Abstract. In this paper, we propose a reasoning model aimed at helping 
to decide on the virtual status of a given situation, from a human point 
of view rather than from a technological one. We first describe how a hu- 
man and his environment interact. The notion of “reality” will be seen 
through this description. Then, we propose a set of possible “cognitive 
deviations” of reality leading to situations of virtual reality. This model 
provides three major benefits to the field of Virtual Reality: first, a global 
definition and a systematic mean of categorizing related situations; sec- 
ondly, the ability to discuss on the virtual status of real situations and 
not only synthetic, computer generated ones; thirdly, a demonstration 
on how the field of Tele-Operation is heavily related to virtual reality 
concepts, and some perspectives on future tele-operation intelligent user 
interfaces. 



1 Introduction 
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2 Reality as a Set of Cognitive Interactions 

2.1 The Mj^C-r Descriptive Model 
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3 What is Virtual Reality? 

3.1 Sources of Virtuality 
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vi w ti i 1 lity tu lly on i t of imit ting p i 1 of 1 nvi 
onm nt. fo w ju t in f ont of not of typ 1 vi tu li tion 

p o 
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3.3 Type 2 Virtuality Sources 

Augmented Reality P viou ly w pok of ugm nt lity in t m of 

ing vi tu 1 to In. w on i on p t ing 

vi tu 1 to t 1 n . n nt p p f om N g o n 

ugm nt lity y t m giv to t op to “ . . . t f ling of t Iking 

to t o j t it If . . . . n t op to look t ook i not only 

giv n t in info m tion out t ook nt t ility to t Ik i tly to 
t ook n k it f w mo qu tion . i x mpl of ugm nt lity 

0 pon X tly to w t w 11 typ 2 vi tu li tion p o . t 

nvi onm nt i not ju t imit t ut it vio i mo i . Mo n mo 

imil p o to u nt y t m .g. in omput i u g y 

w fo in t n t op to i giv n t ility to t oug t p ti nt 

kull n in t nk to vi tu 1 m on t 1 im g of t 

op ting on . 

Released Reality g n 1 pu po of ugm nt lity i n m ly to 

ugm nt o j t iliti p ption info m tion t . imil ly t ount 

p t of ugm nting iliti i imini ing i iliti . i m n t t not 

po i ility fo typ 2 vi tu li tion p o li in t ility to “ 1 1 

wo 1 on t int fo in t n t in ility to v tim to p f om 1 w 
ofpyi t.vllti nl y nin om y t m known 

ugm nt lity y t m . p f t xp ion “ 1 lity . 

om X mpl in V 1 M ( i tu 1 lity Mo ling ngu g ) vi w 
t u n n 1 / i 1 olli ion t tion toggl w Ik/fly n vig tion 

mo . o option lik going t oug w 11 o flying o t no 
pon to 1 ing v 1 p y i 1 on t int . imil ly in v 1 vi tu 1 lity 

vi og m o vi tu 1 mu um vi iting y t m t u njump i tly f om 
on oom to not 1 ing t i w y t t mpo 1 on t int of mov m nt . 

o p o typ 2 vi tu li tion p o in t 1 vio i not 

imit t ut mo i 

3.4 Definition of Virtual Reality 

ving om vi tu lity ou in giv n itu tion in y to p k of vi tu 1 

lity ut not u i nt. u mo 1 of lity f tu t of g nt on 

longing to it own 1 . n ou point of vi w t t m “ lity i nti lly 
o i t wit t notion of g nt 1 . i m n t t 11 w t 

vi tu 1 o not oul follow t mo Itutu ntu ptt gnt 

1 . imil ly t t m “vi tu 1 mo to o wit t g nt in t n 

woul t n lity t t onfo m wit t MffC mo 1 ut 

wit vi tu 1 g nt in t of Ion. on i tion 1 u to p opo 

t following nition 

t u t k itu tion V to x min . i itu tion mu t onfo m 
wit t MrIC mo 1 n will 11 itu tion of i tu 1 lity 
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if n only if on o mo of it g nt i f om t o pon ing 

1 on ( ) n if 11 u g nt (t n 11 “vi tu 1 g nt ) n 

0 t in f om 1 on t nk to vi tu lity on of typ 1 n /o 2. 

in in ny t g nt 1 qui to t m t topog p y 

of t onn tion tw n t g nt mu t p v w 11. on 

qu n t fou typ of int tion i li n y ompon nt 

of ny i tu 1 lity y t m. i point of vi w It oug on ption of t 
t m “int tion i p i li i ommonly g on. 

4 Immersion and Virtual Reality 

mong t 11 on pt n i involv in u nt vi tu 1 lity y t m t 

notion of “imm ion i t mo t wi ly u . n t i tion w ow ow 
t on pt of mm ion t wit on nition of i tu 1 lity n w 

mon t t ow ou mo 1 n 1 u to i on t vi tu 1 t tu of 1 

wo 1 too. 

4.1 Immersion and Mj^C 

Overview mo t w 11 known of imm ion it on in w i t op 
to volv in omput g n t wo 1 t nk to t uit Mount 

i pi y ( M ) t glov . . . n f 1 p y i lly p nt in t i vi tu 1 nvi 

onm nt. 

n u itu tion t um n o y n p ting g nt i in ont t 

wit n w xt n 1 g nt imit ting t 1 nvi onm nt.no to k p o 
nt p y i If ling of p n i 1 imm iv y t m oul on qu ntly 

i onn 1 1 m nt 1 g nt f om i 1 nvi onm nt ut till n 1 p f t 

mo 1 of t um n o y n it “mutu 1 tion wit t imul t nvi on 

m nt (p y i 1 on t int olli ion viul oyf k...). n f om 

point of vi w t m nt 1 g nt oul ompl t ly i onn t f om 

it own p y i 1 g nt n link to ompl t ly vi tu 1 on in t . i i 1 

itu tion n mo li own in gu 2. n t i mo It vi tu 1 it 

u tion i o t in y “ vi ting t info m tion p t Atp n PpT tow 

n imit tion of t 1 p y i 1 g nt. i mo li tion ompli wit ou 

nition of i tu 1 lity in t w ol p y i 1 g nt i vi tu 1 on (it 
i f om t 1 on ) n t i g nt i o t in t nk to typ 1 vi tu lity 
ou 

tup i t t w not ugg ting ( n w o not t ink) t t it 

oul po i 1 to p t min f om oy. utw n ytmp ovi n 

1 nt imm iv f ling to t op to n fo in t n p o u li ti 

ynt ti im g of t o y t n t op to om lik ly to “ li v t 

img() n pttf k()gt Ion. iiwt 

ou mo Ip nt . 
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Fig. 2. Mi^C-i Mo 1 of mm ion 



Limitations t t i tim u nt imm iv y t m f f om ing i 1 
u t y k two impo t nt info m tion p t of t M/^C-i mo 1 
n not ompl t ly i onn t t op to f om i own nvi onm nt. n 

vi tu 1 o g m t o pi y i in ont t wit t 1 g onn v n 

if li V it i t o on t nt t i itu tion i pt 1 only long 
t vi tu 1 g onn o pon to t 1 on . n t i itu tion t info m tion 
p t omp i ing Ape n Pep not ompl t ly vi t qui y on 

nition of i tu 1 lity. 

itu tion in w i t p op io ptiv f k i impo t nt in 

flig t imul tion n ing t t of g vity o 1 tion i i ult to 

i V nt oul p nt fo li ti n ing. info m tion loop 

k App PpT (in i t ont ol g nt) i u ntly ok n. i t t 
ny vi tu 1 lity y t m oul n 1 p op ly 11 int tion ompon nt 

in In ing p op io ption ompli wit t point of vi w of u u 

f t MifC oning mo 1 i on n w nnot oppo t no 

logi 1 gum nt to t o ti 1 mo 1. fo w t ink t t t imul tion 

y t m it ov o V lop tow vi tu 1 lity y t m nt y t im 
p f t u to t nologi 1 on t int . 

4.2 Can Real Worlds be Virtual? 

w V only pok n of ynt ti nvi onm nt until now n impo t nt 
f tu of on mo 1 i to giv u t ility to xt n t fl tion to 1 wo 1 
too. nt nxtp gp w will xpl in w y n ow 1 wo 1 n 
vi tu 1 un t in i um t n n g in w will po ition t t ition lly 

of ugm nt lity. 



m iguou 
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Real Immersion n t p viou tion w It wit imm ion in 
nvi onm nt (“vi tu 1 imm ion ). Now w woul lik to ow t t t 

p t of t imul tion i not n y to v n imm iv vi tu 1 

lity y t m. on i itu tion of imm ion in ynt ti nvi onm nt 

w i i im t p o u ing t x t lity. y gui ton in vi tu 1 

mu um w t op to n n vig t in oom ( nt till olli wit w 11 ) 
n look 1 1 p inting . Now on i itu tion in w i n op to i giv n 
X tly t m int f ( mount i pi y joy ti k . . . ) nt i int ting 
wit 1 ( i t nt) mu um. t i 1 t t f om t op to point of vi w 

t itu tion m in t m . 1 itu tion n 11 “vi tu 1 lity 

u lik in t imul t w p ovi ompl tlynwpyi 1 gnt 

( 1 in t of ynt ti t i tim ) wit it own ont ol n xt n 1 g nt . 

i n w g nt i in vi tu 1 u non of t tu 1 p y i 1 g nt om 
pon nt uppo to p v . Mo ov in t i vi tu 1 p y i 1 g nt 
i tu lly 1 on it f 11 in t t go y of typ 1 vi tu lity on 



info m 


tion 


P t 


A.^pPpj' 
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u vi t t 


m w y in ot 
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t mo 


Ip 


nt 


in gu 


2 i 


till V li p 


nt tion of t i 


itu tion. 
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i w y w 


t ink t 


t t 


i tin tion ynt 


ti / 1 i not 


1 V nt to 


1 1 


itu 


tion ‘ 


Vi tu 1 


lity . 


ognitiv ly p 


king i tu 1 


lity mu t 


not 


t i 


t to omput 




imul tion. 


i point of vi w 


i f om 


t on 


of 


u 


In u 


n 


fo w om vi 


tu 1 lity y t 


m mu t 



on omput g p i fo nyt ing 1 



Augmented Reality ugm nt lity itu tion int ting u 

t y oft n mix 1 n ynt ti ompon nt . in w ju t w t t 1 n 

n un t in i um t n vi tu 1 lity itu tion t qu tion of 

t vi tu 1 t tu of ugm nt lity i . t ink t 1 1 i t tu p n 

on t itu tion 

f t op to in i t ont t wit t 1 n(u tlop ting 

i t nt o ot o vi iting i t nt mu um) t itu tion my tu lly 
mix of In ynt ti imm ion o t in t nk to typ 1 vi tu lity 

ou . n it i pt 1 to 1 It itu tion i tu 1 lity. t woul 

tu lly mo pp op i t to p k of Augmented Virtual Reality in 
1 imm ion Ion i 1 y itu tion of vi tu 1 lity w ju t w. 

ft op to in i t ont t wit t 1 n n t go 1 i not to 

ng t i (lik volving wit t n lu nt gl w i up impo o j t 

onto t 1 n ) w n no long t Ik of imm ion in t op to i 
xpli itly qui to t y in ont t wit i 1 nvi onm nt. i f t n 
o n t omply wit ou nition of i tu 1 lity. w n p k of 

ugm nt lity ut not of i tu 1 lity. 
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5 Examples 

o illu t t fu t ow w nut mo 1 to i on t “vi tu 1 

lity t tu of u o u itu tion 1 t u giv t outlin of t oning 

on two ont i to y x mpl . 

5.1 Example 1 

n t i X mpl 2 n op to i imm in vi tu 1 nu 1 pi nt n i 
giv n t ility to n vig t in t pi nt n p fo m v 1 op tion . 

o pon ing 1 itu tion i t on in w i t op to t in n 

imil nu 1 pi nt. it p t to t MffC-r mo It n w g nt 

follow 

ont ol g nt m in t m in t 1 itu tion. n t op to 
till i w t to o n ont ol t vi tu 1 o y. 

it littl t ining (1 t ining m n tt y t m) t op to will 

1 to u t joy ti k y fl x to mov oun . i impli t t t 

t n ition 1 g nt m in t m in t 1 itu tion w n t y t m 
int f i p f tly imil t y t op to . Moving oun in t vi tu 1 
nvi onm nt oul in om o viou w Iking in t 1 oom. 

xt n 1 g nt om t nu 1 pi nt omput mo 1. i i n 

imit tion of 1 nvi onm nt. t u in f ont of typ 1 vi tu li tion 

p o 

p ting g nt om t um n o y omput mo 1. It oug 

t i mo 1 i not p f t ( .g. only t op to n i vi u lly p nt ) 

t ultim t go 1 i 1 ly to imit t t op to o y w i i typ 1 
vi tu li tion p o 

now V t min t o pon ing 1 long wit t n w 

g nt involv in t itu tion. t u k t v li ity oft g nt o 
ing t t 1 pi tion impo y t Mj^C mo 1. 

op to i giv n ot t ility to mov i vi tu 1 o y n to p iv 
it(tltt n) n p ting t Atp n Apt info m tion p t . 
n w p ting g nt t ility to g o j t in t vi tu 1 pi nt 
n olli ion t tion i impl m nt n n ling o tly t App n 
App info m tion p t 

n V itu tion in w i n w (vi tu 1) p y i 1 g nt i p ovi 
i g nt n it int tion wit t m nt 1 g nt onfo m wit t MffC 

mo 1 qui m nt . Mo ov t i g nt i o t in t nk to typ 1 vi tu lity 
ou . t u in f ont of itu tion of vi tu 1 lity n mo p i ly 

itu tion of “vi tu 1 imm ion . 

5.2 Example 2 

tutk gint X mpl p opo y N g o . u wit t n lu nt 
n w n in li y n tim look t ook t oug t n 

n V lly int t wit it i tly. t u i 1 y in Hi y 
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o t i no ot o pon ing 1 itu tion. y t m o not tu lly 

p ovi ny n w g nt in t op to i xpli itly qui to t y in i 

own nvi onm nt. n w n 1 y on In t t t i itu tion woul n t 

11 i tu 1 lity. u nition pi t ttl tonoft mo 1 

g nt oul pi w i i not t . ow v t i x mpl 

illu tt t ftttw n vvitu lity on in itu tion wit out it 
ing i tu 1 lity itu tion p op ly p king, n giving t “f ling 
of t Iking to t ook it If on titut typ 2 vi tu lity on . om t 

op to point of vi w n o j t (t ook) i vi tu li y mo ifying it 

no m 1 vio . i itu tion long to t t go y of ugm nt lity 

(typ 2) nt o ing to on nition not to i tu 1 lity y t m . 

6 Virtual Reality and Tele-Operation 

n t i tion w mon ttt tt 1 oftlop tion n vi tu 1 lity 

t ongly It n w ow ow on mo 1 n ing out v 1 int ting 
p p tiv on w t oul futu int f fo t 1 op tion y t m 

11 “t 1 op tion itu tion in w i n op to i mot ly wo king 
in i t nt^ o in i 1 . u itu tion u ti i 1 tool to ot 

t n mit o to t m nipul tion vi ( .g. o ot) n iv f k 
f om it. 



6.1 Sources of Virtuality in Tele- Operation 

Immersion t i impo t nt to noti t t mo n mo t 1 op tion y 
t m u vi tu 1 lity int f t t p ption 1 v 1. ft u ing n 
to i pi y t i t nt n t 1 op to p ovi wit M llowing 
t m to g t p ti 1 vi w of t n v n to mov t i point of vi w. n 

u y t m t i i tu lly to p ovi n imm iv nvi onm nt to t 

op to . iving t op to t f ling of ing p nt in t i t nt 

mk u ytmfll into t t go y of vi tu 1 lity y t m u t y 
tu lly impl m nt of 1 imm ion w i w p ov to of 

i tu 1 lity. i on titut t t 1 tion tw n t 1 of i tu 1 
lity nip tion. 



Augmented/Released Reality p o of vi tu li ing t 1 op tion it 
u tion o not top t t imm ion 1 v 1. M ny nt ppli tion v 
own t utility of ugm nt lity on pt ( u p ovi ing ym oli 

info m tion up impo on t vi u 1 1 ) fo t 1 op tion . ow v t 

on pt of ugm nt lity xtnft t njutt p ption 1 v 1 t 

m in i n tw n 1 op tion n imul t op tion i t t in 1 
t op tion u u lly i v i 1 . i on t int 1 to 

V lop t ining y t m w 11 utonomou ( n op fully out) o ot . 

^ What we call “distance” here means spatial distance as well as temporal distance. 
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n u ing t ining y t m t t 1 op to i tu lly giv n n to 

“ V tim in of mi t k . i i f 11 into t t go y of 1 

lity n typ 2 vi tu li tion p o 

6.2 Virtualizing the Mental Agent 

Until now w v mon t t ow vi tu li tion p o n t t 

xt n 1 g nt t op ting g nt n t mo t t w ol p y i 1 g nt. og 
i lly t qu tion of t mining w t vi tu li ing t ot g nt in t 

mo 1 i nil i . i qu tion m k n if on i 1 tiv ly to t 

1 of t 1 op tion. 

Virtual Control Agent y ing utonomou i ion p o in t 1 
op tion y t m w nt n w 1 of vi tu lity w tu lly p ovi n w 
( n vi tu 1) ont ol g nt in t y t m oil o ting wit t op to . 

om Mj^C point of vi w t i i t t tim t t vi tu lity on It 

wit t m nt 1 g nt it If. op to tu lly g t t f ling of ont oiling 

not only n op ting g nt nt n op ting ugm nt wit minimum of 
int llig n t u mo ifying t ommon vio of 1 op ting g nt . i 

i of ( yp 2) ugm nt i tu 1 lity not only y p ption not 

only y tion nt y fl 

Virtual Transmutation Until now w on i two m in itu tion m 
m ion in w i t op to i m nipul ting t nvi onm nt wit “ i own 

n nip tion in w i t nvi onm nt i mo i in i tly 

t oug o ot i tiv . n t i 1 t n impo t nt p o 1 m i to p ovi 

y t m y to 1 n wit t ining p io o t po i 1 . w p 

ntly t t long t op to ont t i tly wit i own o y o 
wit n i omo p i ynt ti v t lik in n imm iv itu tion p io of 

pt tion in y to 1 to u t int f (joy ti k t glov . . . ) 

wit out ving to t ink out it. wit u on t int t ultim t go 1 

of t 1 op tion y t m oul to p ovi vi fo w i t woul 

no mo t ining p io w t op to oul giv o only y moving i 
own o y. on i m ly o ot y p 1 of illing n wing 

op tion . op to i giv n ynt ti vi ion of i m up impo on 

t im g of t 1 n . i w y ( ) ont tu lly noti t p n 

of o ot ut in t g t t f ling of wo king wit i own n . n 

illing op tion in t op to ju t point ng to t illing 

on n t o ot imm i t ly t k 1 ill n op t qu t 

(g ing t ill m y not i pl y in it i not t nti Ip t of t 

op tion). imil ly vi tu lly g ing w n pulling it up to t ol 

woul m k t o ot 1 V t ill g w n p fo m t op tion 

i tly. t i int ting in t i o t ining int f i t t t op to 

only n to know t n tu 1 mov m nt n oing 11 i lif . n 

V yt ing o u if t o ot m t op to n w op ting g nt. 
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i vi tu li tion p o i t u “vi tu 1 t n mut tion fo t op to . 
u tituting o ot to t um n o y i typ 2 vi tu li tion p o in 
t vio of t op ting g nt i ompl t ly ng . 




Fig. 3. Mj^C-d nt ntion t tion 



Intentions Detection n n v n mo ultim t vi w t i 1 itu tion i t 
on in w i t op to o not v to wo k ny mo ! i m n t t if w 
w nt t op to to k p ont ol ov t v nt t im of t y t m woul 

to t t ft po i 1 t op to int ntion n p o utom ti lly. 

il t V y futu i ti vi w of u itu tion i m nt 1 ommuni tion 
tw n t op to n t y t m t i go 1 n till pp o wit n 

“int ntion t tion y t m 9 minimi ing t p y i 1 wo k of t op to . 

o in t n giv nt p n ofnojtint nvi onm nt n giv n t t 

t op to i u ntly moving i m tow t o j t n int ntion of 

g ing t o j t i oming mo n mo lik ly n oul in tu n 

t t . n t m of vi tion f om t MifC mo It y t m ommuni t 

wit t op to t t ig t po i 1 m nti 1 v 1 t t i i tly 

wit t ont ol g nt. i itu tion i mo li on gu 3 n p nt t 

ig t 1 V 1 of vi tu lity tt in 1 w il k ping n op to in t itu tion. 

7 Conclusion 

n t i p p w uilt ognitiv mo 1 of int tion t oug wit on 

pt of lity i n. p opo t notion of “vi tu lity ou im t 

mo ifying 1 ompon nt of itu tion in o to m k t m om vi tu 1. 
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notion of i tu 1 
mo g nt of t mo 
m jo n t 

u mo 1 p ovi 
p ti ul itu tion. 
on pt . 

i t go i tion p o 
to 1 wo 1 . 

u mo 1 mon t t 

i tu 1 lity onn 
futu int llig nt t 1 op 



lity w t n n t vi tu li tion of on o 
1. t ink t t on vi ion of i tu 1 lity t 

of ny 
1 t 



y t m ti w y of 
i will op fully Ip 



i ing on t 
1 ifying t 



vi tu 1 t tu 
notion n it 



i not limit to ynt ti wo 1 nt 1 o xt n 



t 



1 of 1 p tion n 
1 int ting p p tiv on 



n on 
tion o 

It to t 
f mo 
t i p p 
V lop 
i t n 



opinion t 
not li in t 



to w i xt n 
t n p ovi 
tion int f 

m in i tin tion tw n i nt i tu 1 lity itu 
ynt ti o 1 p t of t itu tion ut t in t 

0 pt.il typ 1 vi tu lity ou m inly 

notion of imm ion it m t t typ 2 vi tu lity ou woul 

Ipful in t 1 op tion y t m . ow v t illu t tion it long 
t n to ow t t typ 1 vi tu lity ou u ntly mu mo 

t n typ 2. n in pt on t notion of 1 p tion 

1 m y of om Ip to o t t i im In . 
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Abstract. This paper proposes a solution to the problems associated with 
network latency within distributed virtual environments. It begins by discussing 
the advantages and disadvantages of synchronous and asynchronous distributed 
models, in the areas of user and object representation and user-to-user 
interaction. By introducing a hybrid solution, which utilises the concept of a 
causal surface, the advantages of both synchronous and asynchronous models 
are combined. Object distortion is a characteristic feature of the hybrid system, 
and this is proposed as a solution which facilitates dynamic real-time user 
collaboration. The final section covers implementation details, with reference to 
a prototype system available from the Internet. 

Keywords: distributed virtual environments, network latency, collaboration, 
interaction, multi-user, virtual reality. 



1 Introduction 

Within a distributed virtual environment users interact with objects and state 
change events are generated. These events are normally transmitted to other users 
within the virtual environment across a network. The timing of events can be 
managed using either synchonised or asynchronous clocks. 



1.1 No Clock Synchronisation (Asynchronous) 

In this model, the environment contains no uniform, global time. Each host has a 
local clock, and the evolution of each event is described with respect to this local 
time. Because there is no global time, all event messages are sent without reference to 
when they occurred, and so events are replayed as soon as a user receives them. In 
this way, users always perceive remote users as they were some time in the past. 



1.2 Global Time (Synchronous). 

When using synchronous time, as events are transmitted, they are time-stamped 
according to a global time that is synchronised between all users [1]. When events are 
received by a user, they are replayed at the correct global time. This usually requires 
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the model to be rolled back to the correct time of the event - time must then be 
advanced quickly to the current time, to allow real-time local interaction. 



1.3 Network Latency 

The network connecting users within the virtual environment induces latency of 
information interchange, which causes problems in the areas of user and object 
representation. The following two sections and Fig. 1 describe the problems, and 
suggest that different time management techniques are best for the two areas. 



Time 


View seen by remote user 


t=l 


^ moving along a path (z=0,x=tj. Xa 
'\X^: XjT ,/ (modelled using asynchronous time) is seen 

ij I 2 I 3 7*** delayed by 1 second, Xs (modelled using 

/ synchronous time) is consistent with X. 

Z 


t =2 


y. Local user input causes X to follow a 

X (shown by dotted line), this 

/vvv z message begins transit to remote user. Xa is 

H '2 S 7 * delayed by 1 second, Xs begins to become 

7 inconsistent with X. 

Z 


t=3 


Message about path change is received 
by remote user. X still follows new path. 
Xa still delayed by 1 second, so now the 
V y / r'®"' P^th can be started without any 

l-l >2 T * discontinuity. For the time t=[2,3] Xs is still 

4 / ; X following linear path, so is inconsistent 

^ ' y ^ X. Now Xs has to discontinuously 

jump to X. 



Fig. 1. User positions as seen by a remote user. X is the local user, Xa is the user modeled 
asynchronous time, Xs is the user modeled using synchronous time. The network delay, A , is 
1 second 



1.4 Asynchronous Time for User Representation 

As Fig. 1 shows, user representations will appear discontinuous if a synchronous 
model is used. The asynchronous model, however, does not lead to discontinuities so 
is advantageous for user representation. 



1.5 Synchronous Time for Object Representation 

If asynchronous clocks are used, a local user will always see the past state of 
remote users and objects, as Fig. 1 shows. The model of objects at the local site will 
therefore be different to the model at the remote site. If objects appear in different 
places to different users, the users cannot interact with those objects. The only 
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solution to this problem is to use synchronised time to model objects, so the positions 
of objects and the interaction responses will be consistent to all users. 



1.7 Proposed Solution 

The technique described in this paper proposes a hybrid synchronous/asynchronous 
model, combining the advantages of each method. A global time must be maintained 
between all participants to ensure consistency and allow interaction between users and 
objects. As in an asynchronous model, remote users are displayed at a previous time 
to ensure no discontinuities arise. The technique employs the causal surface, which 
facilitates collaborative interaction through the use of object distortion. 



2 The Causal Surface 

In the SVi-D perception filter [2] [3], objects are delayed by different amounts as 
they move around space. 

Asynchronous model for remote users. Remote users are shown delayed by A 
to remove discontinuities, as in an asynchronous model. Objects are delayed by A in 
the local user’s view when coincident with the remote user. Therefore, objects that are 
being interacted with by a remote user are displayed at the same time as that user and 
the interaction is viewed without discontinuity. 

Synchronous model for local user/ohject Interaction. Objects are displayed in 
real time close to the local user to allow interaction in real-time. 

The variable delay is achieved by generating a causal surface function, A=S(x,y,z), 
that relates delay to spatial position [4]. This surface is smooth and continuous to 
avoid object position/velocity discontinuities. Fig. 2 shows an example surface, 
generated using radial basis functions [2] . 




Fig. 2. The causal surface divides space-time into the various causal regions. This figure 
represents a local user. A, at the position (0,0), and three remote users (the peaks on 
thesurface). The vertical heights at positions coincident with each user represent the time it 
takes interactions from the local user to propagate to that remote user. For example, the surface 
has a value of A=1.5 at user B, showing that it takes 1.5 seconds for information to propagate 
between A and B 
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In space-time it is impossible for information to reach a point under the causal 
surface, so the surface represents the bound on the causality of events occurring at the 
local user. This causality restriction applies to all information in the environment, 
thereby placing limits on the speeds of both objects and users. 



3 Distortion of Objects 

3.1 Distortion in the Real World 

In the real world, every event takes time to propagate and affect remote objects. 
Einstein proposed that all physical phenomena was limited by the speed of light. This 
critical speed limits the propagation of all causal influences, such as changes in 
gravitational and magnetic forces [5]. Although the speed of light limits all 
interactions, the nature of some interactions result in an effective critical speed less 
than the speed of light. Examples of apparent slower critical speeds are the speed of 
sound, or the speed of wave propagation along a spring. Generally it can be said that 
all interactions are limited by a critical rate of propagation, although the critical rate 
may not be constant, and is dependent on the type of signal and on the material being 
propagated through. Fig. 3 shows how a flexible object results in a critical speed. 




Fig. 3. Distortion of flexible object. Two people, A and B, are holding either end of the 
object. A is moving the end back and forth at frequency, f. When A moves to the left (a), B is 
initially unaware of that movement. Some time later (b) the wave will reach B and B will be 
forced to move left. If A now moves to right (c) it will be some time later (d) before B feels this 
motion 

The critical rate can be defined as a propagation time A - the time it takes 
information to propagate through the object. From standard control theory, the 
propagation time will result in a maximum frequency of interaction, /„, above which 
instability occurs, where 

A=1/(4A). (1) 

As the frequency of interaction approaches this critical frequency (in the absence 
of any visual feedback) then each user’s motions will be out of phase leading to 
resonance making co-operative collaboration impossible. A very good example of 
resonance occurring can be illustrated if two people try to use a skipping rope without 
viewing the other person (this object has very low stiffness and hence a discernible 
propagation delay). By looking only at their hands it is possible to initiate a low 
frequency of rotation but it is impossible for higher frequencies. Brief 
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experimentation showed that standing waves occurred at multiples of the resonant 
frequency (an expected result from the control theory [6] ). 



3.2 Distortion in the Virtual World 

The delay associated with propagation of force along objects in the real world is 
derived from length and stiffness of the material. In the virtual world, however, the 
delay is derived from the network. The propagation of force as waves of distortion 
through objects in the real world is therefore analogous to propagation of event 
messages through the network. By showing the force propagation through objects in 
the form of distortion, users are made aware of the delay that is present, and can adapt 
to the maximum achievable rate of interaction. 



3.3 Collaborative Interaction 

As a user changes the state of objects within their vicinity, those state changes will 
be reflected immediately on parts of the object close to that user. As the state change 
messages are transmitted across the network, the state changes initiated by the user 
will be seen to propagate along the object. When the state changes reach the remote 
user, the remote users interaction is taken into consideration, and resultant actions are 
determined [7]. These actions are sent continually as state changes back along the 
object (and also the network) to reach the local user again. This interaction occurs as a 
dynamic closed loop feedback system. 



4 Implementation 
4.1 Object States 

Each object in the view seen by a local user will be displayed at a time t-A, 
obtained from the causal surface function A = S(x,y,z). The causal surface is 
continuous and dynamically changes as users move and as network delays vary, such 
that the value of the causal surface is always greater than the actual network delay at 
each user. Given that A is bounded within the range [0, maxjS}], all the states of 
each virtual object must be defined over the range [f-max{S), t], where t is current 
time. The trajectory of each object state can be implemented as a function or set of n 
functions: 



^ (O = fi(t) t, < t < (2) 

for i = 1, 2, ... n, where t. is the start time of the t function, and < (t-max{S}). 
The functions f.(t) are stored in a temporally ordered list of functions, the most recent 
function being held at the head of the list. A function /J(t) can be removed from the list 
if a function ^(f) exists in the list such that t < (t-max{S}). 

Every defined parameter of each object (the states of the objects) must be 
specified over the interval [t-max{S), t\. The states of object may include, for 
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example, position, orientation, colour, temperature, but all types of interaction 
available to the users must be defined parametrically. A separate list of functions is 
used for each object state. 



4.2 Virtual States 

The previous sections have concentrated on distortion of shape. Object states 
define all dynamic parameters of the object, including all the states that can be 
influenced by interaction. Therefore, as each point on the object is delayed by a 
different amount, all dynamic states of the object will change across its surface. For 
example, if the temperature of an object is increasing over time, a delayed part of the 
object will be at a lower temperature than a part closer to the (local) heat source. 



4.3 Demonstration Program 

A simple demonstration program, showing the use of the causal surface is available 
from the ISRG website. The program allows two users to interact with objects within 
the same world. A simulated communication channel between the users allows 
variable network latencies. 



5 Conclusion 

This paper discusses an alternative solution to prediction for many of the problems 
associated with network latency within distributed virtual environments. Remote 
users, and objects close to them are viewed by local users as they were some time in 
the past, thereby removing state discontinuities, and allowing visualisation of remote 
interaction. Objects close to local users are displayed in real-time, such that 
interaction is possible. Objects are delayed by a smoothly varying amount between 
the local and remote users. Distortion occurs due to the varying delay, and this has 
strong parallels in the real world - state changes are propagated through objects at a 
finite rate. In the virtual world, this rate is derived from the network delay, and the 
distortion allows visualisation of information transfer to allow real-time dynamic 
interaction. 
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Abstract. This paper describes a set of models and related data elements that 
represent physical properties needed by the simulation algorithms in a virtual 
environment. This paper also describes a VRML based editor called the 
SimVRML editor. The data elements represent physical properties used by 
common models that perform general simulation within virtual environments. 
The data element class definitions described in this paper can be used by model 
algorithms and model classes related to the following simulation categories: 
Vehicle operations, ship and submarine operations, aircraft operations, sensor 
operations (electronic and visual), communications, role playing, combat 
operations, industrial, chemical and biological. The SimVRML editor is a virtual 
environment for editing simulation properties used within general virtual 
environments that support physical behaviors. 



Introduction 



This paper describes research into data element definitions and model related 
objects useful for Virtual Reality simulations using the VRML (Virtual Reality 
Modeling Language) specification. This paper also describes a VRML based user 
interface for editing static and dynamic state information for VRML based simulation 
via the Internet or any network in general. 

The concept of VRML simulation is commonly referred to as VRML behaviors. 
The VRML 2 event model, routes and script nodes, in combination with high level 
programming languages, allows for the use of arbitrary simulation algorithms and 
models inside of a VRML virtual environment. The high level languages 
predominantly used for VRML simulation are Java, Javascript, C++ and C. 

The VR system for editing and updating simulation static and dynamic state 
information is the SimVRML editor. The SimVRML editor provides a virtual 
environment for assigning values to physical properties that are used in VR simulation 
models and algorithms. 

The definitions of the data elements operated on by the SimVRML editor are 
designed from an object oriented perspective and based upon Java as the VRML 
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scripting language. The data element definitions and model objects can be directly 
mapped to Javascript or C++ data structures. 

Previous research on physical properties for distributed network simulation 
required by general model algorithms has included the DIS (Distributed Interactive 
Simulation) protocol [ 1 ]. VRML research related to DIS is being carried out by the 
DIS-VRML-JAVA working group associated with the VRML Architecture Group. 
An alternative to DIS based VRML is the virtual reality transfer protocol [2]. The 
research presented in this paper is about VRML simulations that are completely 
generic. It provides an extensible framework for the editing of physically based 
properties used by multifunctional sets of model algorithms. 



Data Definitions for VR Simulation 



This research is based upon the following metaphor. Physically based simulations 
require models and algorithms to implement the simulation. Large numbers of models 
exist whose algorithms involve physical properties that are well understood, 
standardized and can be approximated by a Level Of Detail (LOD) paradigm. Such a 
„simulation LOD“ paradigm allows VR simulation models to use approximate 
algorithms to manage the computational complexity of simulations within a virtual 
environment. Managing simulation complexity is critical when immersed inside a 
VRML base virtual environment that includes multi-user behaviors. 

Consider the following example of the simulation LOD paradigm applied to a 
VR/VE or VRML sensor simulation. In the case of a Radar sensor simulation, the 
coarsest simulation LOD involves the solution of the generic radar equation. The next 
finer LOD for radar simulation involves a more physically based model of ground 
clutter, weather and multipath effects. Finer radar model LOD’s can be generated by 
considering more realistic models of system losses, electromagnetic interference, 
aspect dependant cross sections, antenna directivity, statistical techniques and even 
hostile actions among role playing fanatics. Descriptions of the equations for various 
radar model LOD’s and their associated model properties can be found in various 
sources [ 3 , 4, 5 ] . 

These various radar model LOD’s require well defined properties that need to be 
initialized to realistic values and updated as the VE simulation progresses. The 
Sim VRML editor provides a VRML based interface that addresses the need to 
initialize and update VR/VE physical properties. These properties involve sensor 
models along with a set of models that provide for the simulation of common 
behaviors in a VR/VE/VRML universe. 

The models that have properties defined for the SimVRML editor described in 
this paper are models for vehicles, ships, subs and aircraft motion, along with models 
for communications, sensors, role playing, combat operations, industrial, chemical and 
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biological. The data element definitions for the properties simulated in the above 
models are defined as a series of Java classes. 

The base class for the Sim VRML editor deals with the variables and methods 
needed to manage data extraction, insertion, descriptive names for the JDBC objects 
and pointers to default geometries and model algorithms for a set of user defined 
LOD’s. The SimVRML editor has a class that deals with the variables and methods 
needed to manage data extraction and insertion of dynamic data such as simulation 
time, pointers to the methods and models used in the simulation, along with status, 
position, motion parameters and other common simulation housekeeping data. 

The SimVRML editor operates on many common Objects that are hierarchical 
in structure. Such hierarchical objects consist of data elements describing physical 
properties and also data elements that describe equipment that in turn are described by 
data elements within other subclasses. Thus, a Vehicle object may have a radio object 
and an engine object as components of its hierarchical structure. 

The SimVRML base class and its subclasses have a public variable that is an 
enumerated type. The enumeration’s are simply ASCII descriptions (less than 33 
characters) of the various properties that are shared by the common objects that are 
simulated via the above models. There are also methods that are used to extract and 
insert valuations for the properties in the SimVRML base class and its subclasses. 
Using derived classes and virtual functions and providing an API for the JDBC fields, 
the SimVRML editor can be made extensible. Thus, a bigger and better model 
algorithm that needs more sophisticated properties can be „plugged“ into the 
SimVRML editor. Other designers could then just publish their enumerations and „get 
and put“ virtual functions. 

Subclasses of the SimVRML base class consist of classes related to physical 
properties for the following object categories. Hierarchical high level objects like 
aircraft, ships and submarines, vehicles and people. Low level object categories of the 
SimVRML editor are sensors, weapons, facilities (Architectural objects), machinery, 
structural elements of objects, computer hardware and software, chemical and 
biological. There is also an interaction class that is used by the engagement model and 
to also keep track of static properties that can be applied to current behavior 
interactions between objects. The SimVRML base class has common physical 
properties that are used by all the model algorithms. 



Model Related Data / Implementation for VR Simulations 



In the SimVRML editor system, the number and sophistication of the properties in 
the enumeration public variable in the various classes is driven by the sophistication 
of the model algorithms. The simulation LOD for the SimVRML system is different 
for the various models. 
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The sensor models and the SimVRML sensor classes representing the various 
sensor properties have a highly realistic LOD. The sensor model requires properties 
related to the advanced radar, active and passive sonar, jamming, electrooptical and 
lidar equations. The radar model requires properties for the basic radar equation, plus 
properties related to advance features including electromagnetic interference, ground 
clutter, antenna configuration, variable cross section, countermeasures and multipath 
effects. The radar model produces a signal to noise ratio that is generated from a 
discretization of probability of detection curves versus false alarm rate and number of 
pulses illuminating the detected object. Physical properties are needed for all the input 
parameters of the various LCD’s of the radar equation. Some radar equation 
parameters for low level radar calculations lump multiple physical functions into one 
parameter and approximate the parameters physical value. Realistic radar calculations 
for high level LOD radar models deaggregate the lumped physical properties for 
advanced radar model features. These high level radar models require appropriate 
properties for input into the radar algorithms. 

The sonar model requires properties for input into the active and passive sonar 
equations, sonobuoys, towed arrays and variable depth sonar’s [ 6 ]. Sonar 
calculations depend upon the marine environment as much as they depend upon the 
sonar equipment physical properties. For sonar calculations, the environment is 
divided into 3 sound propagation zones. Environment effects due to transmission / 
reverberation loss properties are arrays of values for various ocean conditions and 
target position relative to the 3 sound propagation zones. The simplest sonar model 
LOD has table lookup properties for good, fair and bad ocean environments. Sonar 
classification models require individual target noise properties for subcomponents of 
an object. Likely subcomponents are machinery (primarily engines and propellers), 
underwater sensors (especially active and towed) and architectural structures (hull). 
Target noise properties are required for various speed values. Probability of detection 
is calculated for individual noise components by dividing the calculated signal to 
noise ratio by a standard deviation of a normal distribution related to the noise 
component and then comparing to a normal distribution with mean 1. The primary 
problem in sonar modeling is generating reasonable physical values for sonar 
environment and target noise. 

The basic Visibility model operates on object detectability properties. Sufficient 
weather effects are built into the VRML specification for very simple visual 
calculations. An advanced visibility model based upon electrooptical algorithms [ 7 ] 
requires calculation of the number of sensor pixels encompassing the detected object. 
Detection probability is generated via an exponential probability distribution that can 
be discretized into a table look up. The table look up function of the model is based 
upon optical signal to noise thresholds for the various categories of objects in the 
virtual environment. If an Infrared or FLIR model is needed then the thresholds 
require temperature dependence factors. 

The Communications model has a moderately realistic LOD. The 
communications model requires properties related to the basic communications and 
jamming equations, which are similar to the radar equations without a cross section 
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factor. Properties related to advanced communications features that depend upon 
clutter, antenna configuration, interference, multipath and terrain are similar to those 
properties needed by the radar equations to calculate similar radar effects. The 
Communications network model has a low LOD to calculate path delays. Path delays 
are calculated by assuming that delay points behaves according to M/M/1 single 
server queue with exponentially distributed service rate and interarrival time [ 8 ]. 
Properties needed by the communications network models are mostly dynamic 
properties related to compatibility of network nodes and communications equipment 
operational status. Such properties reside in the classes used for simulation 
housekeeping. 

The motion model is a low to medium LOD model that requires properties 
related to basic ballistics, navigation, navigation error, along with maximum and 
minimum properties (weight, speed, altitude...). The Flight, ship and submarine 
operations models are similar to the queuing and probabilistic elements of the 
communications network model but require interaction with the motion model, fuel 
consumption properties and specific motion constraints. Fuel consumption 
algorithms require speed and fuel factors for minimum fuel use and also fuel factors to 
interpolate fuel use at speeds other than the minimum fuel usage velocity. Ship and 
submarine operations are similar to aircraft flight operations but with different 
constraints. Motion constraints can be thought of as a set of rules depending on the 
type of simulation object and the model applied to the object. Navigation errors are 
drawn from a normal distribution with mean zero and standard deviation specific to 
the navigational sensor object. The motion model and all the other models require 
system failure calculations. For failure calculations, we have equation 1 where A is 
mean time before failure and B is probability of failure. If B = 0 then the failure time 
is the current time plus some appropriately large constant. 

Failure time = A [(Log(l-rand(x))/(Log(l-B))] + current time (1) 

The engagement model is a combat operations model. The engagement model 
requires properties related to engagement probabilities (failure, detection, acquisition, 
hit) and equipment characteristics. Air related engagements are based upon simple 
probabilistic equations. Other properties related to air engagements and air to ground 
engagements are sensor activation time, sensor scanning arc, lists of target sensors, 
lists of targets and weapon lifetime. Ground based direct fire engagements are related 
to a discretization of the Lancaster equations [ 9 ] based upon lookup tables. The 
lookup tables are indexed based upon the characteristics of the objects engaging. 
Ground engagements between hierarchical objects, such as groups of vehicles are 
indexed based upon group characteristics. The results of the lookup table searches are 
passed to a simple probabilistic model. Ground based indirect fire, which is 
engagement between objects separated farther than a user specified threshold, is 
modeled via a probabilistic model. Indirect fire assets such as artillery are also 
included in the direct fire model. 
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Engagement damage is related to a set of probability factors for failure, 
deception and repair along with defensive and offensive characteristics. Damage 
factors are calculated for the following categories of an object: structural, fuel, 
equipment, and operational status. Damage factors are simple multipliers that are 
compared to thresholds that enable/disable/ or repair/destroy components of the 
object. 

It is very hard to generate standard properties for people. This is due to the 
complexity of people and diversity of opinion on how to approximate human 
behavior. In the Sim VRML system, basic person physical properties would be 
compared to environmental constraints that would be managed by a basic simulation 
housekeeping process. Beyond constraint checking, such social models would be 
relegated in the SimVRML system to managing verbal communication within the VR 
scene and input / output of commands within the VR simulation. 

In the SimVRML system, models for structural, chemical, biological processing 
and any other remaining behavior functions are queuing models augmented by very 
low level algorithms. These low level algorithms use the current set of properties as 
thresholds that either disable / enable the object or switch from one geometry LOD to 
another. As an example, if the melting point is exceeded then the object consisting of 
a solid geometry would be replaced by an indexed face set of the melted object. 
Another example would be the failure of an object that would be used by a damage 
model to affect the state of other objects in the virtual environment. 



Model Interaction and Object Interrelationships 



In order for the extendable features of the SimVRML editor to function optimally, 
the interaction of the model algorithms and the object interrelationships need to be 
defined. The level of complexity of the interrelationships of the object physical 
properties in an extendable and hierarchical VE generated by SimVRML class 
schemas is just the local complexity of the data / model interaction. If the virtual 
environment is distributed or if control is shared in a local virtual environment then 
several factors influence the complexity of the object / model interactions. The 
primary factors are time, transfer / acquisition of control, transfer / acquisition of 
editing rights to the properties and control / acquisition of model algorithms and 
model LOD’s. Time synchronization depends upon agreement on how to broadcast 
time, when to declare a time stamp valid and how to interpolate time stamps. Much of 
the preliminary agreement on time is present in the VRML 97 specification [ 10 ] as it 
applies to nodes and simple behaviors. 

In a complex VRML simulation one method of agreement about the overall 
virtual environment within a distributed simulation is to adopt the rules inherent in the 
DIS specification. The SimVRML editor adopts a hybrid approach consisting of the 
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DIS rules and methods to override the rules or disallow overriding based upon model / 
object „rights“. The SimVRML base class has methods to extract data sets from 
multiple object schemas and to set the structure of the data requirements based upon 
the model algorithms and model interactions. 

The results of the model and data interactions requires that model execution, 
data storage and data extraction be synchronized. One way to affect global control of 
various model and physical property interactions is to treat the system as though it was 
a parallel simulation environment. The housekeeping management scheme chosen for 
the Virtual Environment generates a set of simulation semaphores. These simulation 
semaphores are used to make sure that data is extracted or stored at the correct times 
and that models for the physical behaviors execute in a realistic sequence that 
produces common sense results. 

The SimVRML system is designed to be an extendable system with user defined 
models and physical properties added to a „base system" of models and physical 
properties. In this case there can be multiple simulation semaphore sets that can be 
activated or deactivated based upon virtual environment rights possessed by 
controlling avatars or simulation umpires. Suppose you have a VR simulation that has 
user defined models appended to a base collection of models and their associated 
physical properties along with their defined interrelationships. If you start with a 
realistic simulation semaphore set that produces realistic simulation results, how do 
you produce a realistic simulation semaphore set for the extended VE containing the 
user defined models. One way to satisfy the requirement that the extended system 
produce realistic results is to require a sanity check on the results. The simplest form 
of sanity check would require the simulation state to be outside of obvious error states. 
Eurther sanity checks would compare the simulation state for conformation to test 
parameter limits. Such sanity checks require well defined test simulations within test 
virtual environments. 

Consider the model and physical property interactions for simple linear motion. 
Eirst, consider scale. Aircraft and ship positions are best stored in geodetic coordinates 
while ground motion is best served by Cartesian or Universal Transverse Mercator 
(UTM) coordinates. Motion inside a building is most likely to be Cartesian. Motion 
over several kilometers is probably best represented by UTM coordinates. Next, 
consider weather and environmental conditions. Object states are needed for daylight 
calculations. Navigation properties have to be combined with object motion 
characteristics. Motion requires awareness of the surrounding objects that exceeds 
simple collision detection. Motion requires collection of data regarding the visibility 
status of objects and the properties of the collecting sensors. Thus, data must be 
extracted from multiple object classes. Motion requires execution synchronization 
between the motion, navigation and detection algorithms at the minimum. If objects 
are interacting in a manner that alters the properties and geometry of the interacting 
objects then interaction algorithms must execute. Results of physical object 
interactions need to be propagated to the other motion algorithms under the control of 
motion semaphores. 
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One solution for the simplification of synchronization of model algorithms is to 
separate behaviors that must be event driven and simulation cycle based. Cycle based 
algorithms can be applied to all objects in a simulation subject to pre defined 
conditions. Processing VR behaviors via a Simulation cycle and then broadcasting the 
results to the VR objects affected by the behaviors solves some of the problems 
introduced by object and model interactions. Processing via simulation cycles 
introduces other problems. One problem is related to a requirement that user actions 
should be able to trigger model execution asynchronously. Event based algorithm 
execution requires awareness of all other active and possibly planned events and a set 
of rules and rights to umpire simulation logjams. Very simple model LOD’s can be 
implemented so that events can be reliably precomputed and queued. Complex model 
LOD’s with significant numbers of property and model interrelationships require 
constant updating of the event queue. 

Consider the detection phase of the motion calculation in the event based 
portion of the VR simulation. The object in motion would have an instantiation in the 
VR world along with an instantiation of the detector such as eyes or a night vision 
goggle. There would also be an instantiation of an object that represents the model 
that is being executed to process the detection. The methods of the „detection model“ 
object would extract the relevant physical properties from various object classes and 
spawn events to manage and localize the effects of the interactions between the 
models and physical properties. Users can make their own constructors / destructors 
and methods for model objects. Thus the user can attempt to manage the problems 
posed by model and property interrelationships by incorporating management of the 
interrelationships in the objects representing the models. 

One paradigm for managing the relationships between physical properties and 
also the simulation model algorithms is to use standard object oriented design (OOD) 
methodologies such as OMT [ 11 ] and Shlaer-Mellor [ 12 ]. OMT and other OOD 
methodologies attempt to generate correct relationships between physical properties, 
events and simulation model algorithms. By applying OMT or other such OOD 
methodologies to the Sim VRML base classes and simulation model algorithms, a 
reasonable management scheme for the property / model interrelationships of a 
baseline SimVRML system can be produced. 

By using OMT and other equivalent OOD methodologies, users planning to 
extend the baseline SimVRML system can minimize effects of their user defined 
physical properties and models upon the SimVRML baseline system. Similarly, 
baseline rules for simulation cycle based execution within the SimVRML system can 
be produced by standard methods of semaphore design. The Shlaer-Mellor 
Methodology has been analyzed mathematically to examine the question of 
correctness of OOD for event driven simulations [ 13 ]. 

The SimVRML editor can be used to initialize the state of a Virtual 
Environment containing realistic general behaviors. This initialization procedure is 
performed by editing the physical properties that are dynamic. The default VRML 
description of a virtual environment contains a wealth of geometry, position and VR 
sensor information. Beyond geometry and standard VRML sensors and interpolators, 
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what do you put in the housekeeping classes to attempt to produce a simulation 
semaphore set that provides accepted results under test conditions? The housekeeping 
properties should include enough simulation state data to keep track of object states 
used by the models, simulation rights and the defined semaphore sets. In addition to 
the geometry inherent in VRML nodes, these housekeeping properties keep track of 
simulation commands and data that can be used to map the model algorithms to the 
physical properties. 

In addition to managing data and model relationships, housekeeping properties 
are used to store descriptions of multi-user interactions and pointers to static physical 
properties for interactions. These interactions are different from the model and 
physical property interactions described above. Multi-user interactions describe 
behavior effects between objects that are actually engaging in simulated virtual 
behavior. One of the fundamental multi-user interactions between objects in a virtual 
environment is information communication. Housekeeping properties are used to track 
communications text and dynamic data communications within the virtual 
environment. One function of the Housekeeping properties is to track the simulation 
state of speech simulations and to track spoken simulation command input. By 
keeping track of simulation LOD’s in the housekeeping properties, the correct set of 
static physical properties can be extracted by a generic virtual getModel method. By 
combining low level state information with high level interaction and model LOD 
state information a virtual environment simulation can truly become more than the 
sum of its parts. 



Sim VRML Editor Interface 



The SimVRML editor was created using 3D Website Builder [ 14 ] and Cosmo 
Worlds [ 15 ]. Although there is still no integrated development environment for 
VRML behaviors that is affordable, acceptable components are now becoming 
available. 3D Website Builder is used to create the geometry of the SimVRML editor 
interface. Cosmo Worlds is used for polygon reduction, VRWE/VRML sensor 
creation, VRML routing and scripting within the script node. VRML scripting was 
done in Javascript [ 16, 17 ] and Java [ 18, 19 ]. 

The SimVRML editor interface is a compromise between the goals of a 3D 
anthropomorphic VR/VE/VRML interface and the cardinal goals of webcentric virtual 
reality: speed, speed and more speed. A fully anthropomorphic virtual reality behavior 
editor interface could be realistically composed of 3D subcomponents constructed 
using thousands of polygons. For example, the average power property used by the 
sensor model algorithms could have a realistic model of a generator unit with 
clickable parts in an anthropomorphic version of the editor interface. 
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The design paradigm applied to the SimVRML editor interface reduces the 
anthropomorphic aspect of the interface to texture maps. Photo images or rendered 
images of 3D models representing the various simulation model properties are applied 
to 3D palettes constructed using simple solid primitives. For properties representing 
high level hierarchical subcomponents of an object, such as equipment, this texture 
map strategy works very well and is illustrated in Figure 1 . 




Fig. 1. Equipment Palette SimVRML Interface for Hierarchical Object 



For more esoteric and abstract properties with no isomorphism’s between 
anthropomorphism and algorithmic functionally, a different reduction strategy was 
developed. In the case of abstract properties, an anthropomorphic texture map or 
maps, representing an aggregate of related properties was generated. Hot spots 
representing the individual properties of the aggregate texture map are used by the 
editor to edit property values. Consider the basic radar equation. Sets of texture maps 
of generic radar properties are placed near to a texture map of a text version of the 
generic radar equation. The equation texture map is subdivided and overlain upon 3D 
primitives. Subdivision is necessary to avoid browser environments that do not 
support the VRML image texture or material nodes transparency features. Since the 
Virtual Reality Modeling Language is extensible, extending the SimVRML editor 
interface by users who have added extra model properties is a simple task. Extending 
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the SitnVRML editor interface is primarily a case of image processing and 3D 
geometry building 

User input is through a series of knobs and sliders. This input strategy is made 
necessary due to the lack of a keyboard sensor in the VRML specification. The 3D 
Input interface for both sliders and knobs is similar. The interface consists of a VRML 
model of a slider or knob constructed from primitives and indexed face sets, along 
with a 3D indicator and a 3D control. 3D indicators are used to display the current 
value of the property. The user modifies the property values using the movable 
portions of the slider or knob by clicking or dragging. The user can also use the 3D 
control to edit property values by clicking on subcomponents of the control, such as 
arrows. The radar equation texture map palette is illustrated in Figure 2. 




Fig. 2. Aggregate Texture Maps for Radar Parameters in Sim VRML Editor 

Sliders, knobs, controls and indicators have their own standard set of 
„SimVRML editor properties“. These interface properties are properties for minimum 
/ maximum input values, data type of values, current value, type / size / color of 3D 
text for display of the current values, scaling factors and position / orientation relative 
to the slider / knob interface components. Just as the simulation models require a set 
of standard properties, there is a need for some sort of standard set of input properties 
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for virtual environments. The geometry of the input widgets is up to the input 
interface designer. The SimVRML editor interface implements what I consider a 
minimal set of input properties. Figure 3 illustrates scalar data types. Arrays and other 
structures could be displayed by duplicating and offsetting VRML group nodes 
representing the various types of scalar data types displayed via geometry primitives 
and VRML text. 

The SimVRML editor has an option for 2D operation. This option is enabled 
from within the 3D virtual environment and is based upon Java applets. The 2D option 
has a SimVRML applet for each of the various object classes. The 3D user input 
virtual environment is also illustrated in Figure 3 




Fig. 3. Slider and Knob Input Interface With Control (arrows) and Indicator 

One of the most important considerations related to editing properties for VR 
simulations is the realism of the data values. The majority of properties in the 
subclasses of the SimVRML base class have well known published reference sources. 
Commercial instances of the various abstract objects are particularly well 
documented. Military and adventure instances of simulation objects are not well 
documented. Potential sources for such adventure data are user guides for various 
game simulators, documents published by Jane’s Information Group and 
manufacturers advertising brochures, repair manuals and user’s manuals. Many 
physical properties can be obtained through standard scientific reference works such 
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as those published by the Chemical Rubber Company. Fascinating and generally 
correct sources of physical properties are newsletters and publications of hobby 
organizations such as the American Radio Relay League. 



6 Future Work 



The SimVRML editor is used to edit static characteristics for physical properties 
used by VRML behavior models. The next obvious extension is to create a 
SimVRML layout editor to operate on the variables and methods of the dynamic 
properties. As an example, consider a Yacht with no engine, antennas, 
communications or navigation equipment. Using such a layout editor, the user could 
drag and drop engines, radio and other equipment onto the VRML yacht. The 
equipment would be from palettes containing thumbnails created by querying the 
VRML simulation behavior database. 

The Model algorithms and the methods for extracting data need for the 
algorithms have to be implemented through a SimVRML behavior editor. There is 
also a need for a SimVRML command editor to dynamically input parameters into the 
VR behaviors. One obvious enhancement to the various model algorithms is to use the 
fact that a VlUVE/VRML simulation is intrinsically in 3 dimensions. The current 
damage model simply decides on the fate of a particular VR object. An enhanced 
damage model could also change the geometry of the object using the 
create VRMLFromURL method or by inserting and deleting nodes. Physically relevant 
3-D dials and indicators would replace two-dimensional textual status boards. In 
addition a set of OMT diagrams for the baseline SimVRML editor and related models 
have to be generated. 
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Abstract. We are developing the S'canST'racfc virtual environment (VE) 
using multiple cameras. The Scan&Track VE is based upon a new method 
for 3D position estimation called the active-space indexing method. In 
addition, we have also developed the geometric-imprints algorithm for 
significant points extraction from multiple camera-images. Together, the 
geometric-imprints algorithm and the active-space indexing method, pro- 
vide a promising and elegant solution to several inherent challenges facing 
camera-based virtual environments. Some of these are: (a) correspon- 
dence problem across multiple camera images, (b) discriminating mul- 
tiple participants in virtual environments, (c) avatars (synthetic actors) 
representing participants, (d) occlusion. We also address a fundamen- 
tal issue: can virtual environments be powerful enough to understand 
human-participants. 
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2 The Scan&Track System 
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th p rti ip nt um i r nt po ition th h p o p rti ip nt ilhou tt 
on th m r -im 1 o h n . h r or w xp t th om tri -imprint 
t to V ry rom on r m to noth r p n in upon th h p o th urv 
in th m r -im rom on mom nt to noth r. 

h V impl m nt n automatic r ur iv plittin 1 orithm whi h n 
u ully 1 with r itr ry (multipl ol in ) urv . i ur 3 how th 

xp t orr pon n tw n om tri -imprint o on po to noth r. i - 

ur how thr m r -im ( - ) o th m po n th ir om tri - 

imprint . or t il o thi 1 orithm r in 39 . 
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Figure 1: Postures express 

emotions 




Figure 2 : The Correspondence 
Problem 




Figure 3: (a) Spread out pose. Five 
geometric imprints, (b) Dancing 
pose. Seven geometric imprints. 




Figure 5: (a) Scan-line algorithm 

for automatic extraction of the 
contour, (b) Geometric-imprints 
algorithm using the automatically 
extracted contour 




Figure 6: The relationship of 
points P and Q changes depending 
upon the view as the cameras move 
around them. 



234 234 234 




1 (d> 1 (e) 1 (£) 



Figure 4: Geometric-imprint points of three 
camera images for the same pose, (a-c) three 
camera-images for the same pose, (d) 4 (e) 4 and 
(f) 5 geometric-imprint points. 




Figure 7 : The relationship of 
the point on a slice remains 
same for a variety of planar 
views in the same hemisphere 
w.r.t. to the slice. 
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Only on houl r point in i ur h n i ntifi th om tri - 
point, h impl m nt tion i ntifi fiv ylin ri 1 n -point on thi ompl x 

urv in i ur n . n i ur w h v u th m po or th im 

ptur rom thr m r . Noti th t 11 th i nifi nt point m rk on to 

fiv in i ur - r ommon in th thr fi ur . On xtr 1 ow-point i 1 o 
om tri -imprint point in i ur . n our impl m nt tion w h v m inly 

hown th tip o ylin ri 1 h p how v r th pi wh r ol in o ur 

n 1 o th om tri -imprint point xpl in in 39 . hown in i ur 
imil r topolo i 1 urv n r t imil r om tri imprint t. On th 
om tri imprint h n o t in it i mu h i r to fin orr pon n . 
Not th t om tri -imprint point m rk 1 in i ur - orr pon to th 
m 3 -point (ri ht-hip). imil rly point m rk 2 3 orr pon to th 

tip o th ri ht h n h po ition n th tip othlthn rp tiv ly. 

Not th t th ixth point n i r y u in th tiv - p in xin 

m tho xpl in low. h im -imprint o th ixth point in i ur 

m t h with ny im -point in th oth r two m r -im will r ult 
in no 3 point orr pon n u in th tiv - p in xin m tho . hu 

th orr pon n pro Iminth n rkytmi implifi . i ur 5 
how om tri -imprint o t in y u in n automatic n-lin 1 orithm 
0. olor im r t http://www.cs.uccs.edu/~semwal/VW98. 

4 The Active-Space Indexing Method and Associated 
Results 

4.1 Camera Calibration, Space-Linearity and Over- Constrained 
Systems 

h r h V n m ny tt mpt to tim t th 3 motion o th p rti ip nt 

with minim 1 num r o point u or m r - li r tion or x mpl thr 

point r u i nt u t y 11m n 6 . ntly th r h v n m ny 

tu i wh r fiv or mor point r u or t r o-m t hin n r tin 

nov 1 vi w 20 21 . o t o th y t m whi h u minim 1 num r o point 

r u u lly un r- on tr in in th n th t ny rror in m r - li r tion 

or m r ori nt tion tim t m y r ult in v r r i tr tion n ur y 

rror 13 . 

n impl m ntin th tiv - p in xin m tho w u v r 1 point 

urin pr pro in n rt ptil3-t tru tur u in th point . 

Our motiv tion to u v r 1 point i to u ivi th tiv - p o th t only 

w point r u lo lly to tim t th po ition urin tr kin imil r 

to th pi -wi i n or ur n ontour . n y t m u in minim 1 

num r o point lo 1 li r tion i u n th r or it r t n un r- 
on tr in y t m 13 . h non-lin rity n tr kin rror r mor pro oun 

u to th u o n un r- on tr in y t m. 

n our ppro h y u ivi in th 3 tiv - p into m 11 i joint 
vox 1 or 3 - 11 w u only m 11 num r o point whi h r th v rti 
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o th 3 -vox 1 or tim tin 3 po ition in i th t vox 1. n thi w y only 
w point tim t th 3 po ition. h m jor v nt o u in 1 r 
num r o point i th t th non-lin rity u to m r li r tion i r u 

n p rti ul r w n um th t lin r motion in i th m 11 3 -vox 1 

p will 1 o proj t lin rly on m r -im . hu th to m r - 

i tortion r voi .or nr t tim tion within th 3 - 11 or vox 1 r 
po i 1 thi umption 1 o How th u o lin r int rpol tion within th 
3 - 11 or vox 1. 

4.2 Estimating Depth Using Multiple Cameras and Slices 

on i r th proj tion o two point n in i nr 6. p n in upon th 

vi wpoint n multipl m r in th two point n th proj tion 
o th two point h n p rti ul rly th ir r 1 tion hip h n w mov 

rom 1 t to ri ht rom im pi n to im pi n vi pi n L (L t) 

( nt r) n (ri ht) hown in i nr 6. h p ti 1 in orm tion tw n 
th two point i lo t. ow n w in r th t point Qa n Qb r tu lly 

th m point in 3 p N xt on i r two point n on pi n r- 
li n th m to multipl m r hown in i ur 7. Not th t th 

p ti 1 r 1 tion hip o proj tion o point n in 11 th pi n in th 
m h mi ph r w.r.t. th pi n r li r m in m . hu it i mu h i r to 

1 with point in pi n r- li . n ition zoom-in or zoom-out loo not 

h n th r 1 tion hip o th vi i 1 point . n p rti ul r w.r.t. to two lin LI 

n L2 on pi n r itr ry zoomin o th m r or h n in th ori nt tion 

o th m r in th m h mi ph r hown in i ur 8 o not h v 

n t on thi r 1 tion hip. n p rti ul r point r m in to th ri ht o 

proj t -lin LI n point to th 1 t in oth th proj t -im . oth 

point r ov th proj t -lin L2 xp t . n now xt n th i 

to m ny lin . ri o lin p rtition th li into v r 1 i joint ri - 11 
in i ur 9. i ur 9 1 o how m r im whi hi in pi n r li 
t n n 1 pointin li htly upw r . h r i p r p tiv orm tion o th 
lin y t th r 1 tion hip o point n i on i t nt with th pi n r li 
n th 2 11-in x o oth point i m . 

11 thr m r r pi in u h w y th t th y r in th m h mi- 
ph r w.r.t. th tiv - p li.Ltl2n 3 th proj tion o 

3 point on th 1 t nt r n ri ht m r im r p tiv ly. hi i 

hown in i ur 9. 11 th tripl t ( 1 2 3) n imprint- t or point . 

r 12n 3rth2 pix 1 oor in t o th proj tion o point 

on th 1 t nt r n ri ht m r -im r p tiv ly. 3 point i vi i- 
1 rom multipl m r th n th lo tion o th imprint o th 3 point in 

multipl m r -im n u to tim t th 3 po ition o . iv n th 

im -imprint (1 2 3) th active-space indexing method fin th 3 - 11 or 
vox 1 ont inin th point . n our pr nt impl m nt tion th tripl t r 
provi y th u r. 
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4.3 Preprocessing to Create Active-Space Indexing 

ultipl pi n r li n now t k or tim tin th pth in orm tion. 

h tiv - p li o upy p 11 n tiv - p volum or imply 
n active-space, h li ivi th tiv - p into t o i joint 3 - 
11 or vox 1 . m r -im or v ry li r pro on y on . urin 

pr pro in on m r i pi u h th t th vi w-norm 1 i p rp n i ul r 

to th whit - o r u or our xp rim nt . h oth r two m r r to th 
1 t n ri ht o th fir t m r hown in i nr 9. h v ho n 12 y 12 
ri p tt rn wh r ri int r tion r hi hli ht y m 11 1 k ir 1 . h 

ri -p tt rn o upi 55 m y 55 m p on whit -or. h qu r o th 
ri i 5 m y 5 m. h whit - o r i phy i lly mov n r or y thr 
m r t th m tim . i ht u h r or in r ult in i ht li or v ry 

m r . h int r- li p in i 10 m. h whit -or n th ri -p tt rn 

on it r vi i 1 rom 11 th thr m r . h tiv - p volum i u o 
55 m y 55 m y 70 m. h tiv - p i 1 r non h to tr k th n 

upp r o y o th p rti ip nt or two t o xp rim nt w h v on u t 

or. 1 r r tiv - p n ily on tru t yuin Ir rpnl 

in t o th m 11 r whit - o r u or our xp rim nt . 11 th 12 y 12 

ot on i ht o th li r hown in i nr 11 - or 11 th thr m r . 

u o th p limit tion w o not xpl in th pr pro in 1 orithm 

h r . or t il pi r r to 38 . 

4.4 Position Estimation 

0 tim t th lo tion o 3 point iv n it imprint- t(l 2 3)whv 

impl m nt th ollowin 1 orithm. h tiv - p in xin m h ni m fin 

th 3 lo tion on th imprint t tripl t. h tripl t in our impl m n- 

t tion i provi y th u r y u in mou -pi k on th r p tiv m r - 

im . or X mpl i th tip o th no i vi i 1 in 11 th thr m r 
im or our xp rim nt th n w n li k on th tip o th no in 11 th 
thr im to o t in tripl t ( 1 2 3) or th no . hi i hown in i nr 

10 - wh r 1 2 n 3 r th proj tion on th 1 t nt r n ri ht 

m r or th point . 

iv n im -imprint (1 2 3) orr pon in to 3 -point w r h 
11 th i ht li n fin th vox 1 ont inin point . u to p limit tion 
w h V not xpl in th t il o thi 1 orithm hr. t il r in 38 . 

i ur 10 - how th r ult o our impl m nt tion u in ix imprint- t . 

orr pon in 3 vox 1 r 1 o hown y vi u lizin th m onn t 

k 1 ton fi ur . h v hi hli ht th orr pon in point in th im in 

1 ur 10 n 10 . noth r li htly i r nt to point i hown in i ur 

10 . i ur 10 how th p rti ip nt po y u in n in-hou ynth ti 
tor V lop in our 1 or tory. h v 1 o t t our tiv - p in xin 

m tho y u in two i r nt m r ttin thu r tin i r nt to 

11 n in xin mhnim nhvotin imil r r ult . 
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Figure 8: Projection of two on the respective camera-images, 
points on two planes. 




Figure 10: a-c: Selecting six image-imprints on 
the middle, left and right camera images. 

(d) associated 3D-cells connected by a simple 
skeleton, (e) Another skeleton representing a 
different set of six points, (f) A simple 
synthetic actor mimics the pose of the participant. 




(a) Center camera 




(b) Left camera 




(c) Right camera 

Figure 11: Active-space points 
for the three images. 
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Cor-2respondeiice 





Figure 14: World of active-spaces 
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h t hniqu work u th r i nou h hi t in th proj tion o th 
point u to th m r po itionin th t th pth i rimin tion i po i 1 . 

nl th thr m r n 1 r i nti 1 or th ri i v ry wi it i hi hly 

unlik ly th t mor th n two li h v th m r orm y 1 2 n 3. n 

irn to our m tho thi woul m n th t th r olution o th li tiv 
p n to fin r or m r n 1 n to h n . t n on In 
th t iv n thr imprint-point uniqu 3 11-in x n oun . in i- 

t or olor -im r t http : //www. cs .uccs . edu/~semwal/VW98. 

5 Block Diagram of the Scan&Track System 

h loki rmoth ytmi hown in i nr 12. h n r k y t m 
i n out i lookin -in ytml. t on it o (a) the Correspondence y t m 
n (b) the Active Space Tracking y t m. 

hr r our u - y t m to th orr pon n y t m 

(i) Contour extraction m r to n lyz n n outlin o th il- 

hou tt i xtr t rom th im . L t ontour 1 2 n 3 r pr nt th 

hum n- ilhou tt rom thr m r -im ml m2 n m3 r p tiv ly. 

hi i w 11 r r h r o omput r vi ion. h v v lop n-lin 

1 orithm or utom ti xtr tion o ontour 0 30 31 . 

(ii) Significant points determination xpl in rli r th om tri - 

imprint i xtr t y th om tri -imprint 1 orithm. 

(iii) Significant points correspondence and matching h om tri - 

imprint o on ontour r orr pon to th om tri -imprint o ontour 

rom oth r m r -im pturin th m po . h out om o th m t hin 

1 orithm i to provi n imprint- t tripl t ( 1 2 3) or v ry point in 

th om tri -imprint xtr t rom th m r -im 

(iv) Scene, color, and previous poses database h n in orm tion 
olor o om tri -imprint point n pr viou po r to u or r fin- 

in th orr pon n n m t hin o th om tri -imprint ro multipl 

m r -im 

h tiv p r kin y t m i u to tim t th po ition o point 
iv nth tripl t ( 1 2 3). h tiv p r kin y t mh th ollowin 
two m in u - y t m 

(i) Active-space scanning and tracking system xpl in rli r 

w u th tiv - p in xin m h ni m whi h Ilow u to t rmin th 

lo tion o 3 point iv nit om tri -imprint ( 1 2 3). t thi tim w 

p i y th orr pon in point on multipl - m r im u in mou -pi k . 

n utur th point woul automatically n r t y th orr pon n 
y t m hown in i ur 1 . r inv ti tin nov 1 p tt rn ( . . h k r 

or p tt rn ) to utom ti lly fin th ri -point on th pi n r li urin 

pr pro in . 

(ii) Synthetic actor and/or avatar display in th tiv - p in- 
xin m tho w h v n u ul in po itionin hum n-mo 1 or our 

i ility xp rim nt . n utur n v t r or ynth ti hum n tor i xp t 
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to r pli t th motion o th p rti ip nt. n utur w pi n to ompl t ly u- 
tom tth n rkytmotht pronin 3 nvironm nt n 

tr k n r pr nt y n v t r. hi i th o 1 o our propo r - 

r h. 

Multiple participants: loxptth n rkytmto work 

wh n multipl p rti ip nt r in i th tiv - p hown in i nr 13. 

r n o multipl p rti ip nt n th ir im r r n h 11 n in pur ly 
vi ion- y t m . or x mpl it i hi hly lik ly th t th om tri -imprint 

or h m r im will not h v n i nti 1 num r o point v n thou h 

m r r vi win th m n . hi oul u to o In ion y oth r 

p rti ip nt in th . xpl in rli r orr pon n i mu h i r or 
th n r k y t m th tiv - p in xin provi v In 13 in or- 
m tion. hown in i nr 1 m h- plittin m y n ry wh n multipl 

p rti ip nt r pr nt 32 . 

Ratio-theory: o r olv r m -to- r m orr pon n w 11 orr - 

pon n ro multipl im o th m n w pi n to u th on pt 
o r tio th ory rlt to oyprt. hi hin th ratio theory n 

t un r too y im inin t ilor m nr m nt. h xtr miti o on- 

tour n rlt ily i w t rt with th im n ion o th p rti ip nt 

hown in i nr 1 n 1 . rom on r m to noth r th 1 n th o th 

o y- xtr miti o th p rti ip nt n not h n . L t u um th t th 

om tri -imprint o two po h v n o t in hown in i nr 1. 

now n to orr pon point with point 1 point with point 2 n o 

on. h n th point r onn t i. . 1 to 2 to 3 n o on. n i nr 1 th 

urv 1 n th LI tw n xtr mity n orr pon to th urv In th L2 
tw n n no on. n p rti ul r it n i th t th r tio L1/L2 
woul pproxim t ly qu 1 L3/L . o th orr pon n pro 1 m r u to 
m t hin n fin in t fit in m 11 r h- p p n nt upon th 

num r o om tri -imprint point on urv rom m r -im 

Double verification: h n r k y t m How double verification 

or V li ity o orr pon n . or x mpl i it i oun th t tripl t ( 1 2 3) 

r mthw n lovriyit ymt hin th olor o 1 2 n 3 rom 

th im . th y r i r nt th n w o not h v m t h. n ition w 
n h V V ri ty o oth r m ur . or x mpl i th tim t po o n 

V t r i r ti Ily i r nt th n th po in th pr viou r m th n it mi ht 

in i t orr pon n rror uni th p rti ip nt i movin xtr m ly t in 

th 

Avatars and Human Actors: in virtu 1 nvironm nt r xp t to 
popul t y oth V t r or hum n orm whi h r pli t th mov m nt 
o p rti ip nt n virtu 1 ynth ti tor who utonomou mov m nt i i- 
r t y omput r 23 19 . n th n r k y t m th tion o th p rti - 

ip nt r to tr k n m pp to th ynth ti hum n orm 33 3 35 36 37 . 
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6 Expected Significance of the Scan&Track System 

i tru o ny vi ion- y t m th i rimin tion p ility o our y t m 

i limit y th phy i 1- iz o th pix 1 in th m r -im . ow v r th 
y t m Ilow th u o multipl m r n thu multipl in i n mor 

ur t pr i tion o pth. On w h v t rmin 3 - 11 or vox 1 v In 

n n pproxim t lo tion th n w oul utom ti lly u noth r to 
m r or mu h lo r n pr i look tth3-p n rth3 -point, 

n oth r wor th tiv in xin m tho n ppli twi or mor . h 

fir t tim it woul u to fin hi h- r in in x n 1 t r w woul fin 

mor pr i in x y u in tho m r whi h h v th mo t t il vi w o 

th li n n r y r . h in orm tion out mor t il vi w woul 

r or urin pr pro in th m r rr n m nt o not h n 

urin tr kin . hu w r u tin th u o lo 1 n lo 1 m r . 

Not th t on m r oul lo 1 or on point n oul li t lo 1 

or om oth r point in 3 p . tiv in xin i xt n i 1 n m n 1 

to h r w r impl m nt tion; p i 1 purpo hip n in or po ition- 

tim tion 1 ul tion . n ition i r nt li -p tt rn oul v il 1 or 

low n hi h- n u r or utom ti pr pro in . h active-space indexing 

mechanism mp3 point in i th tiv - p to orr pon in proj tion 
on th m r -im 

mu t n w r on o viou qu tion how woul w 1 with th itu tion 

wh n th m r i pi with u h n ut n 1 th t 11 th li proj t 

lin . n thi itu tion ri -lin my r ly vi i 1 or v ry ut m r 

n 1 n th li will proj t lin wh n th m r -im pi n i pr- 

p n i ul r to th li . not th t it i hi hly unlik ly itu tion m r 

r placed by us n w n n ur th t non o th li r p r 11 1 to th 

vi w-norm 1 o ny m r . 

h on r 1 t qu tion i n w till tr k 11 o th tiv p with 

th r tri tion th t m r n not pi t ut n 1 w.r.t. li . h 
n w r i th t 11 th t il o th t r ion n otin yuin mr 
li htly w y n lo ly zoomin on th r o int r t. n ition i or 

om r on th m r n not mov th n th ori nt tion o th li 
n 1 o h n urin pr pro in to r t n w tiv - p in xin 

m h ni m. t i thi r om to h n th ori nt tion o m r n /or li 

whi h m k th tiv - p in xin m tho v r til . or omm nt r 1 t 

to r olution ur y r pon iv n ro u tn r i tr tion n o i ility 
r in 38 . h n r k i xp t to accurate n h v xtr m ly 
hi h resolution u v r 1 m r n pot nti lly zoom-in n provi 
hi h r olution tiv - p ir . h tiv - p in xin i on t nt 

tim wh n th num r o li n lin r on t nt. h responsiveness o 

th n r k y t m woul 1 o p n upon th tim it t k to 1 ul t 

imprint-point rom th m r -im . hi n to urth r inv ti t 
ut w not th t th orr pon n pro 1 m i mu h implifi u to th 

tiv - p in xin n th om tri -imprint 1 orithm. t i li v th t 

th orr pon n u-ytm(iur 1) will th k y to r 1-tim int r tion 




The Scan&Track Virtual Environment 



77 



n houl impl m nt in h r w r mu h po i 1 39 . h tiv - 

in xin y t m i xtr m ly robust it n Iw y fin 3-11 or iv n 
imprint-point within th tiv - p . in in 38 th n r k 

y t m u m r -im whi hr tt r in r olvin th swimming pro 1 m 
n ount r in virtu 1 nvironm nt . irtu 1 o j t t n to wim u o th 
li ht V ri tion in tim tin th 3 po ition o th tr k -po ition. u 

0 th tiv - p in xin th m orr pon in point on th im will 

Iw y pro u th m r ult n th r or th wimmin pro 1 m i xp t 

to on i r ly r u 

urin pr pro in m r n zoom in n out n pi p n - 

in upon th u r wi h. ultipl ur ntrkth m3 p upon 

th ir own hoi o li n ot p tt rn . N w m r n u to mor 
tiv - p ir . 

h tiv - p in xin m tho i uniqu n n w n it i n tt mpt to 
olv th orr pon in pro 1 m ro multipl m n i 1 o n w m tho 
to tim t 3 po ition. ontour xtr tion m tho in th propo 1 fin tho 
point on th hum n o y whi h r mo t vi i 1 n th r or r mor 

lik ly to pr nt n -point in ontour xtr t rom m r -im 

Occlusion: nth n rkytmwplnto look into p t vi o- 
r m to r olv or t 1 t know th t th om thin whi h w vi i 1 or 

1 not vi i 1 now. n thi th propo n r k y t m will ontinu to 

tr k tho n -point whi hrviil(..h n t). h point i thi 
th t th n r k y t m i ro u t th om tri -imprint h n rom 
r m -to- r m . 



7 Can VEs Really Understand Humans 



h n r k i xp t to hi hly i tri ut 
r 1 n r k tiv - p phy i lly i tri ut . i ur 
tiv - p . Ithou h tiv p n r itr rily 1 r 

to th hum n-p rti ip nt i provi no tr kin n 

tiv - p .in w xpt vrl hum n-p rti ip nt to 
thi hi hly i tri ut it i ppropri t to k ( ) n 

r ul th t urin hin ( ) ( ) n r lly un 

p rti ip nt 



y t m with v- 
1 how vrl 
nou h priv y 
on out i th 
int r tin in 
mor pow- 
r t n th hum n 



h V th ollowin o rv tion (i) 1 r nti lly mor 

pow r ul th n th urin hin mo 1 u o th non-lin rity intro- 
u y hum n-p rti ip nt . y llowin multipl hum n to int r t in th 
phy i lly i tri ut m nn r r xp t to h v in non-lin r w y 

u hum n h vior i nti lly non-lin r. (ii) Ithou h r non- 

lin r n th r or xp t to mor u ul th n in lin with 

hum n-int r tion it i our opinion th t my not 1 to completely 

un r t n th human participants. Ithou h w n not un r tim t th 
hum n- r in w h v impl y t omp llin r um nt hum nr t in 

non linear abstraction. oo x mpl o non-lin r tr tion n oun 
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wh n w tr V 1. ow m ny o u r lly inv t ny tim thinkin out th pi n 
flyin oon w o r th pi n w r Ir y m kin pi n to wh t 

w will o wh n w rriv t our tin tion. n oth r wor w h v Ir y 
tr t th tu 1 tr V 1 (w um th t w will rriv t th tin tion 
ly). o 1 rly t t our o rv tion m y How u to p r orm non- 

lin r t k how v r th y m y not un r t n th hum n-p rti ip nt n 
1 to pr it with rt inty wh t th hum n-p rti ip nt w nt to o t th 
V ry n xt mom nt. 



8 Conclusions and Final Comments 
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th t point in on t nt tim . h m tho provi p -lin rity within th 3 - 
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fin r ot p tt rn. 

h propo n r k y t m i u ul or i tri ut . ultipl 

ur ntrkth m3 p upon th ir own hoi o li n ot 

p tt rn . N w m r n u to mor tiv - p ir . h 

n rkytm n u orpronl tiv - p in ront o th u r 
r n y pi in thr m r t th top o th monitor, h pr pro in 

1 orithm i impl n n ily utom t.h n rkytm n 

1 o u Ion with oth r xi tin tr kin t hnolo i . m r r 

xp t to mount on th w 11 n out o th w y o th p rti ip nt th 

tiv - p in xin m tho i uit 1 or hum n- nt r ppli tion . 
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Abstract. In this paper, I will show an example of a VRML 2.0 interface for a 
portable device. CyberGlass uses a “video recording metaphor” to examine a 
3D virtual world. The critical problem is to change a viewpoint in the virtual 
world based on the movement of a portable device. We have solved this 
problem using data obtained from a CCD camera. Using a CCD camera to 
detect motion enables us to develop a compact device. We developed a 
prototype of CyberGlass to provide an intuitive access to 3D world. 



1 Introduction 

Given the explosive growth of the Internet and progress of Internet technology, it will 
be necessary to develop technologies that promote various types of access to content. 
Existing access styles are based on HTML and JAVA. VRML 2.0 is a comparatively 
new format for content publishing, which enables us to represent information as a 
collection of 3D mobile graphical objects. Unlike text-based schemes, this type of 
representation can present information which has not been explicitly coded by the 
producer. Even the relative locations of objects can be meaningful. If objects move, 
information will change over time. Thus, we need to discover an application domain 
where VRML 2.0 is an attractive alternative. 

We are attempting to identify a plausible context in which we might naturally 
access a 3D virtual world. This will help suggest a style of content publishing 
appropriate for VRML 2.0 moving worlds. One plausible scenario is to use VRML 
2.0 information in a place for which a 3D model is available. Suppose we are visiting 
the Louvre museum and its corresponding 3D model is available on a portable device. 
We may access appropriate information for each work of art at the appropriate 
moment. It would be useful if the view displayed on a screen of the device matched 
an image being observed. This is because the view allows us to find relevant 
information quickly. 

To make this scenario realistic, we need to solve two problems: one is to determine 
the location we are looking at and the other is to detect motion of a device. Lor the 
first problem, we can assume that infrastructure support will provide us with a global 
position. Lor example, NaviCam uses bar-codes pasted on target objects for which we 
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can retrieve information (Rekimoto, 1995). We are currently focusing on the second 
problem, namely, detecting a device’s ego motion. 

As described above, our final goal is to record our trajectory in a visiting place and 
to replay it using a VRML 2.0 browser with an appropriate 3D model of the visiting 
place. Through our work towards this goal, we have recognized that ego-motion 
detection itself provides a good user interface for VRMK 2.0 browser. In general, a 
gyroscope is better hardware for identifying an ego motion of devices such as head- 
mounted displays (HMD) in virtual reality environments. We chose a CCD camera, 
however, mainly because we do not want to add more hardware to a small computing 
device for simplicity. Again, our final goal is to use a portable device in a real world 
to record the changes of our viewpoint in a visiting place. A CCD camera could be 
used to detect a global landmark. Thus, our challenge here is to see how well we can 
solve our problem using a CCD camera only. 



2 CyberGlass: Design Issues for Portable Interfaces 

We designed CyberGlass for users of portable computers, which are, in general, 
smaller, slower, and have less storage than desktop systems. In order take full 
advantage of these computers, we Identified the following design issues: 

- intuitive operation for examining the world: CyberGlass is a user interface 
designed to help us examine the world. When we look around the world, we 
normally turn our head or move our eyes. In a desktop environment, however, 
because of this restriction in the position of the display, that type of motion is 
impossible. Instead, we change our viewpoint by using pointing devices such as 
mice or touch sensors. When we carry a device with a display, the restriction 
disappears. We can move it to right or left while looking at its display. This is 
typical in video recording. We move a video camera so as to keep the things we 
want to record in the center of the display. Since we are already familiar with the 
action of filming a video, we can use this action as a metaphor for the user 
interface. We shall call this the “video recording metaphor.” In fact, the main idea 
of CyberGlass is to capture images in a 3D virtual space as if through a video 
camera or a water glass in the real world. Note that this is not unusual in VR 
environment, but we are trying to apply the same type of idea to portable 
computers using computer vision techniques, rather than gyroscope. 

- inexpensive movement detection: One of characteristics of portable computers 
is their ease of use - we can use them anywhere and anytime. Nevertheless, they 
have a serious deficiency. In general, to save battery consumption, portables have 
less CPU power and less memory. For this type of computer, simpler algorithms 
are preferable in order to reduce computation time. With respect to ego-motion 
detection, several computer vision algorithms are available. One is the optical flow 
algorithm. Several studies are based on optical flow (e.g. (Horn and Schunck, 
1981) (Sundareswaran, 1991) (Barron, Fleet, and Beauchemin, 1994)). One 
particular example of the optical flow algorithm is shown in (Ancona and Poggio, 
1993). This algorithm is said to be well-suited to VLSI implementation. Therefore, 
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compared with other optical flow algorithms, this algorithm is worth considering 
for portable computers. Optical flow computation, despite its complex calculation, 
can never reduce errors to a negligible level. If we can limit the working 
environment, we can expect that simpler methods such as frame difference could 
work well. Simple frame difference is much less expensive than optical flow 
algorithms. Fortunately and strikingly, in our experiment, the simple frame 
difference algorithm works very well when target images have a clear contrast in 
color depth a condition that we can generally expect around picture frames in 
museums. Therefore, we decided to use simple frame difference for a prototype 
system while we continue to investigate the other algorithm for further 
developments. 

- approximation of rotation angle: Movement of a device is perceived as linear 
movement, even if we rotate the device. We are interested in rotation here. 
Therefore, we need to project linear movement to rotation. 




Fig. 1 A Prototype of CyberGlass. 



3 Implementation 

A prototype system of CyberGlass was developed based on the SONY CCD camera 
MC-1 and TOSHIBA portable computer LibrettoSO (Pentium 75 MHz, 32MB). The 
CCD camera is arranged so as to match its direction with user’s gaze. A picture of 
this system is shown in Fig. 1. This system consists of two software modules: a 
motion detection module and a 3D virtual world’s viewpoint control module. A block 
diagram is shown in Fig. 2. Hereafter, I will explain how this system works. 
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Fig. 2 A block diagram of CyberGlass 



3.1 Motion Detection Module 

The image captured by the CCD camera is stored in a frame buffer. After grabbing 
two successive frames, we perform an optical flow calculation. As I mentioned above, 
the method used is a frame difference algorithm. We can calculate a center of gravity 
for changes of all the pixels at time t by comparing two serial frames of images from 
a CCD camera. Let us call the center of gravity (x, y). After the next frame is 
received, we perform the same calculation to obtain the center of gravity at t +1. We 
call this (x’, y’). From these centers of gravity, we can obtain (Ax, Ay), where Ax = 
x’- X, Ay = y’- y. 




Fig. 3 Approximation of rotation angle of a device 
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3.2 3D Virtual World’s Viewpoint Control Module 

Once we have obtained the movement of the center of gravity (Ax, Ay), the 3D 
virtual world’s viewpoint control module can rotate the viewpoint in a 3D virtual 
world by the following angle 0: 

e = tan’((AP*L„„)/(P*d)). (1) 

where AP is either Ax or Ay in pixels and P is either the bitmap width or height in 
pixels. is a distance in meters corresponding to P and d is a distance in meters to 
a target object in front of the device (See Fig. 3 in detail). The result is then 
represented on the display. 

As described in the introduction, our final goal is to use a portable device in a 
visiting place. Therefore, d must be a correct distance from a CCD camera to a target 
object. In a real environment, by adding a special sensor hardware (e.g. an ultrasonic 
sensor) to the CyberGlass, it might be possible to obtain an exact distance d. In the 
case of a VRML 2.0 browser, however, we do not have any target. D, in this case, is 
the distance from our eyes to a virtual referential object in a virtual space. Thus, we 
need to set d to a constant. 



4 Characteristics of an Experimental System 

We conducted experiments with the CyberGlass equipped with both a CCD and a 
gyroscope for comparison. Fig. 4 shows traces of horizontal and vertical movement 
of the viewpoint according to the movement of a CyberGlass. The gyroscope 
provides almost the exact angle from an initialized position, whereas the CCD uses 
approximation calculated by equation 1. Even so, it should be noted that the 
approximated moving angles have nearly the same tendency as of the gyroscope. In 
particular, the approximated moving angles successfully return to 0 degrees whenever 
the gyroscope says that it is 0 degrees. 

Unfortunately, when we consider both the horizontal and vertical axes, and we 
move a CyberGlass only in horizontal direction, a problem arises as shown in Fig. 4 
(b). Although we did not move it in vertical direction, the angle approximation 
module automatically accumulates erroneous movement caused by noise. As the 
result, the angles went down towards the ground. We can avoid this by a forced 
solution. Namely, a CyberGlass can suppress updates of angles of either direction by 
detecting which direction has a major movement. If we need more degrees to be 
detected at one time, we will need to consider a more complex algorithm presented in 
(Roy and Cox, 1996). 
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5 Conclusion 

We designed a VRML 2.0 interface for a portable device and developed a prototype 
system, called CyberGlass. This system is designed based on a “video recording 
metaphor”. Namely, we can look around a 3D virtual world as if we were looking 
around the real world using a video camera. The current implementation has been 
tested in an environment where lighting is relatively moderate and objects are evenly 
distributed all around the environment so that any movement can be detected 
according to the motion of the handheld device. 





Fig. 4 A Trace of HorizontalA^ertical Movement. 
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One frequently asked question is why we do not try to detect forward or backward 
movements in this interface. It may be interesting to note that, according to our 
forward/backward movement with CyberGlass, the viewpoint inside a 3D virtual 
world changes. Nevertheless, 1 believe that, with respect to forward/backward 
movement, it would be better to use a push button interface rather than to change our 
standing position. One reason is the limited space available for the user to move. The 
other reason is that we are familiar with the metaphor of “driving a car” using an 
accelerator. 

This is a report of work in progress. In future work, we would like to integrate this 
system with NaviCam so as to determine a global position of CyberGlass in order to 
produce a platform for more advanced applications. 
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Abstract. The user interfaces in virtual environments depend on the work task 
given by specific applications and the adaptation to the human senses. This 
paper describes a method for work task analysis and selection of interaction 
devices for the design of user interfaces in virtual environments. The method 
takes into consideration degrees of freedom in the interaction process and the 
human senses. Characteristic is the systematic process for the designer. The 
method will be described and the results in an experimental environment 
discussed. 



1 Introduction 

Over the past few years, Virtual Reality (VR) has grown from an experimental 
technology to an increasingly accepted standard part of the business process. VR 
offers significant benefits as a problem-solving tool in various applications. It gives 
the user the feeling of being a part of the application to see and examine products, 
plans or concepts before they are realized [1]. 

The starting point for problem-solving in Virtual Environment (VE) is choosing 
the technological configuration for a VR system. It is mainly the application, which 
drives the need for particular technology requirements. These requirements are the 
computer hardware, VR software and the configuration of interaction devices. 
Especially the adaptation of interaction devices has a close influence on the human 
performance in problem-solving processes. 

Erom the beginning, VR technology has produced a broad variety of interaction 
devices. Some, like force-feedback systems, for use in specific work fields and others 
for more universal use. Therefore the designer has no easy choice. The result is, that 
in expert interviews the user-interface of VR systems is rated unsatisfactory because 
of its user-hostility [2]. 
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2 Conditions 

One characteristic of VR technology is multi-modal communication between a user 
and a Virtual Environment. A wide variety of VR interaction technology for 
perception and manipulation in VE is available for use in practical applications. They 
differ in performance, information throughput rate or user-friendliness. Eor example 
the visual sense can be served by a common monitor, a head-mounted display (HMD) 
or a CAVE™-system. The choice depends on the necessary degree of perception and 
the work tasks in VEs. To balance both factors is the job of the designers of the user 
interface. VR applications commonly disregard human factors. Most designers have 
not examined the requirements of psychology and perception [3]. 

The main requirements for the interaction devices are driven by work tasks for 
system and application control. The required interactions are so-called necessary 
interactions in VE at configuration and run time. The task of the designer is to fulfill 
these necessary requirements with the available interaction devices and to take into 
consideration the human factors for perception and manipulation processes. Several 
researches have reviewed the important ergonomic issues in VR. But it is rare to find 
ergonomic guidelines for designing VEs and especially the configuration of 
interaction devices. The goal is to gain a maximum in human performance for a given 
work task. Experiments show, that the terms of interaction, presence and interaction 
devices are very important issues in human performance [4]. Sheridan stated that the 
determinants of sense of presence are; the richness of sensory stimuli, simultaneous 
viewpoint changes with head movements, the ability to handle objects inside the 
virtual environment and the dynamic behavior of the moveable objects [5], [6]. A 
breakdown of sense of presence can occurs because of the weight of the HMD, time 
lags or poor balanced configuration of interaction devices in VE. The result is a 
reduce in human performance. To gain an optimum in performance a method for 
configure interaction devices has to take into consideration the factors of perception 
by given requirements. 



3 Method 

The configuration of interaction devices has to take into consideration the 
requirements of the work tasks (necessary interaction of the user), the degrees of 
freedom (DOE) of the interaction devices in VE (possible interactions) and the 
combinations of interaction devices [7], [8]. The necessary interactions of users will 
be separated from the specific work tasks of system and application control and 
classified by the method of Ware-Mac Kenzie [11]. Here the interactions will be 
classified by DOEs. Multiple DOE devices such as 3D or 6D trackers and glove input 
devices are excellent choices for multiple DOE tasks such as viewpoint or object 
placement by gestures. Similarly, 2D devices are well suited for 2D tasks such as 
drawing or pointing on a CRT. The same classification of the interaction devices 
allows a first assignment of interaction devices to the work tasks by DOFs. By the use 
of grading and comparison the final combination of interaction devices will be chosen 
by taking the allowed combinations of interaction devices with their mutable 
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combinations into consideration (Fig. 1 shows the methodical steps to find an 
optimized combination of interaction devices). 

(1) Classification and grading of necessary interactions for controlling and observing 
the VR system based on DOFs. 

(2) Classification and grading of necessary interactions for the application in the VE 
based on DOFs. 

(3) Classification and grading of the interaction devices based on DOFs. 




Fig. 1. Method for task-analysis and selection of interaction devices 

Problems arise, when using interaction devices for tasks that do not have the same 
number of DOF. For instance, specifying the 3D position and orientation of an object 
with a mouse is difficult because one can only modify two DOFs at a time. 
Conversely, using higher DOF devices for lower DOF tasks can also be confusing if 
the interaction device is not physically constrained to the same DOF as the task. 

(4) Grading of the combination of interaction devices with input/output functions. 

Not all combinations of interaction devices are meaningful. For instance the use of 
a computer mouse in combination with a FIMD make no sense. 

(5) Choice of interaction devices by the use of the grading and comparison. 

The choice of interaction devices begins with the comparison and grading of input 
and output devices with the necessary interaction for each DOF. The result is a table 
of interaction devices with a grading depending on system and application relevant 
work tasks. Further the tables will be used to build up matrices with the combinations 
of input and output devices for system and application work tasks. A first rough 
ranking will show, how use-full each interaction device is for a specific work task 
separated for system and application work tasks). Because not all combinations are 
useful, the matrices have to be compared and valued with the possible combinations 
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of interaction devices, described in step 4. The result is a ranking, which interaction 
device fits best the given system or application requirements. The separate ranking for 
system and application work tasks could lead to an overlap of two different work 
tasks to the same human sense. This is sometime very disturbing for a user. For 
instance, printing system information in the view field of the user can be very 
disturbing in an immersive environment. Such overlaps can be solved by the mapping 
one function to another human sense. Here, system information can be presented as 
acoustic signals or as a speaking voice. 



4 Experimental Environment 

The method will be evaluated in an experimental environment for virtual 
prototyping [12]. A VE has to be designed to fulfill design, validation and training 
requirements in safety engineering. Safety systems are very complex and the 
development a difficult task for design engineers [9]. A lot of rules and laws have to 
be regarded. Mechanical and electrical know-how is necessary for the technical 
design, ergonomic and psychological know-how for the control design. Most 
accidents occur because of errors in the control design. Missing ergonomic aspects 
lead to high workload and low concentration. The test environment for the virtual 
prototyping of safety systems consists of a robot working cell with two drilling 
machines and two separate working stations. A worker has to interact with the system 
by programming or reprogramming the robot system, by exchanging the storage 
systems and by the repair of male-functions of the robot system, the drilling machines 
and the working stations (Fig. 2 shows the geometrical model of the robot working 
cell with the safety system components). 




Fig. 2. Geometric model of the experimental environment 
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5 Results 

The process of virtual prototyping consists of an order of work tasks with different 
tools. Each tool is well adapted to the problem-solving process [10]. In a VE the user 
controls these tools by the use of interaction devices. The use of interaction devices 
for problem-solving is not constant but differs from the work task and the DOF the 
device in VE. 

The method for work task analysis and selection of interaction devices has been 
tested in the experimental environment. The first step is to define the work tasks for 
VR-system and application control and observation. Table 1 and 2 show the work task 
separated into the required DOFs, input and output interactions. The grading of the 
tasks is a subjective ranking with 5 levels of importance. 

Table 1. Classification and grading of required interactions for system control and observation 
in virtual environments based on degrees of freedom 
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Table 2. Classification and grading of required interactions for application control based on 

degrees of freedom 
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This method guarantees, that important work tasks will be mapped to high ranked 
interaction devices (Table 3). Classification by DOF of the interaction devices is 
shown in Table 3. The process is the same as for the work tasks before. The grading 
ranks how suitable the interaction device is for the required immersion and presence 
in the VE. 
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Table 3. Classification and grading of interaction devices based on degrees of freedom 
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The classification of interaction devices will be rated by the highest possible DOF. 
The use of interaction devices with a higher DOF than the linked work task is 
possible. All DOFs of a dataglove are for instance hardly ever used. Further is it 
possible to combine different interaction devices with lower DOF than required form 
the work task. Not all combinations are however useful. Table 4 shows the 
combinations of input and output devices with their subjective grading. 

The final selection of the interaction devices will be done by grading and 
comparison. The steps are as followed: 

(1) To select interaction devices, the designer has to combine and value the 
interaction devices with the necessary interactions of the work tasks for system 
and application control and observation. The results are tables with ranked input 
and output devices sorted by system tasks, application tasks and DOF. 

(2) As not all combinations of interaction devices are useful, all interaction devices 
have to be valued with the results of Table3 (mutual influence of combinations). 
The result are rankings for all interaction devices sorted by system tasks, 
application tasks and DOF. 

(3) The last step is a check if there is an overlapping of two different work tasks to 
the same human sense. If there is a collision, one work task have to be mapped to 
another human sense. 

For a virtual prototyping VE in safety engineering the described method leads to 
the following combinations of interaction devices (Table 5). The result is a rank list 
for the interaction devices sorted by the DOF, system (sys) and application (app) 
related requirements. 

The VE for virtual prototyping is a immersive environment based on HMD and 
stereo speaker or headphone as the main output devices. The main input devices are 
the dataglove or flying joystick combined with a tracking system. An exoskeleton 
would add tactile and force perception to the VE. The rank for input devices shows 
that immersive environment is not very useful for the system relevant tasks . To solve 
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this problem a second person should guide the person who does the validation. This 
second person controls the environment with conventional input devices. The 
monitoring of system relevant information is possible and helpful in immersive VE. 

Table 4. grading of mutual influences between interaction devices 
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The ranking of the devices with DOF 1 and 2 in Table 5 shows a correlation. This 
is not surprising because the exercises can be solved with non VR related devices. 
DOF 6 and n show a significant difference. In this cases the exercises are related to 
spatial perception and action. The tendency to VR related devices is reasoned by the 
application related exercises for virtual prototyping in safety engineering. 
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Table 5. Rank of interaction devices for system and application control 




6 Conclusion 

The use of the method for task-analysis and selection of interaction devices have 
been tested in an experimental environment at the Fraunhofer IPA. Therefore a VE 
has been designed for the specific requirements of virtual prototyping work tasks in 
safety engineering. Design, validation and training work tasks selected work tasks. 

The method takes into consideration the classification of work tasks (system and 
application relevant), classification of interaction components and the mutual 
influences of combinations of interaction devices. Its uniqueness is a straight forward 
process to select interaction devices depending on immersion and physiology of the 
human senses. 
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Abstract. In this paper, we present the results of a human factor stndy 
aimed at comparing the effect of stereoscopic versns monoscopic viewing 
on human tracking performance. The experimental paradigm involved 
tracking and grasping gestures toward a 3D moving object. This exper- 
iment was performed using different frame rates (from 28 frames per 
second (fps) down to 1 fps). Resnlts show that monoscopic viewing al- 
lowed stable performance (grasping completion time) down to 14 fps. 
Stereoscopic viewing extended this stabilty to 9 fps, and decrease task 
completion time by 50 % for frame rate under 7 fps. We observed that 
stereoscopic viewing did not much increase performance for high frame 
rates. 
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Fig. 3. xp im n vi u nvi onm n 
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Fig. 4. xp im n on gu ion. 

4 Discussion 
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Trial 



Fig. 7. Mean task completion time versus trials using monoscopic viewing 
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Fig. 8. Mean task completion time versus trials using stereoscopic viewing 



6 Conclusion 

V inv ig 0 0 opi vi wing on um n ing/g 

ping mov m n ow 3 moving viuoj oi ngpi m 

o m n w oun noo iiyi n wn2pn 

1 p o mono opi n o opi vi wing, ow v o opi vi wing 

in p o m n y 50% o m ow n 7 p . i ugg 

o opi i p y mo u u ow m 






106 



Paul Richard et al. 



References 

1. A.P. Georgopoulos, J. Kalaska, and J. Massey, “Spatial trajectories and reaction 
times of aimed movement: effects of practise, uncertainty, and change in target 
location”. Journal of Neurophysiol., 46:725-743, 1981. 

2. C.G. Atkeson and J. Hollenback, “Kinematic features of unrestrained vertical arm 
movement”, Journal of Neuroscience, 5:2318-2230, 1985. 

3. J.A Thomson, “How do we use visual information to control locomotion ?” Trends 
in Neuroscienc es 3, 247-250, 1980 

4. J.A Thomson, “Is continual visual monitoring necessary in visually guided locomo- 
tion ?” Journal of Experimental Psychology: Human perception and performanceO, 
427-443, 1983. 

5. V. Ranadive, “Video Resolution, Frame-rate, and Grayscale Tradeoffs under Limited 
Banwidth for Undersea Teleoperation”, SM Thesis, M.I.T, September 1979. 

6. M. Massinimo and T. Sheridan, “Variable force and visual feedback effects on tele- 
operator man/machine performance” Proceedings of NASA Conference on Space 
Telerobotics, Pasadena, CA. Vo.l, pp:89-98, January 1989. 

7. Y.-Y. Yeh and L.D. Silberstein, “Spatial judgments with monoscopic and stereo- 
scopic presentation of perspective displays.” Human Factors, 34(5), 583-600, 1992. 

8. D. Regan, K.I. Beverley, “Some dynamic features of depth perception” , Vision Re- 
search, Vol:13, 2369-2379, 1973. 

9. W.N. Kama and R. DuMars, “Remote Viewing: A Comparison of Direct Viewing, 
2-D and 3-D Television, AMRL-TDR-64-15, Wright-Patterson Air-Force Base, 1964. 

10. P. Rupert and D. Wurzburg “Illumination and Television Considerations in Tele- 
operator Systems”. Heer, E. (Ed.), Remotely Manned Systems, California: California 
Institute of Technology, pp. 219-228, 1973. 

11. L.A. Freedman, W. Crooks and P. Coan, “TV Requirements for Manipulation in 
Space”. Mechanism and Machine Theory, 12, 5, 425-438, 1977. 

12. R.L Pepper, D. Smith and R. Cole, “Stereo TV Improves Operator Performance 
under Degraded Visibility Condition” Optics and Engineering 20, 4, 579-585, 1981. 

13. R.L. Pepper, R. Cole, E. Spain and J. Sigudson, “Reseach Issues Involved in Apply- 
ing Stereoscopic Television to Remotelly Operated Vehicles” Proceedings of SPIE, 
4 02- Three- Dimensional Imaging, 170-173, 1983. 

14. S.R. Ellis, W. Kim, M. Tyler, M. McCreevy and L. Stark, “Visual Enhancement for 
Telerobotics: Perspective Parameter”. IEEE 1985 Proceedings of the International 
Conference on Cybernetics and Society, pp. 815-818, 1985. 

15. W.S. Kim, S. Ellis, B. Hannaford, M. Tyler and L. Stark, “A Quantitative Evalua- 
tion of Perspective and Stereoscopic Display in Three- Axis Manual Tracking Tasks” 
IEEE Transportation Systems, Man and Cybernetics, SMC-17, 1, pp. 16-72, 1987. 

16. VPL Research Inc., “DataClove model 2 Operating manual”. Redwood City, CA, 
1987. 

17. Polhemus Navigation Science Division, Space Isotrack User’s Manual, Colchester, 
VT. 1987. 

18. Hewlett-Packard, “Starbase Display List Programmers Manual” , 1st Edition, Jan- 
uary, 1991. 




Interactive Movie: A Virtual World with 
Narratives 

yoh i k tsu oko os n k shi O hi 

Mi t r tio o i tio s s r or tori s 

ik ri i ik o or k oto 619 

{nakatsu.tosa, ochi}@mic . atr . co.jp 
http : //www.mic . atr .co.jp/ 



Abstract. 


t 


r ti 


o 


i s 


r 


s 


t o 


i t 


t 


i 1 


ts 


i t 


r tio 




iliti s i 


to 


o i s. 


i t r ti 


o i 


s 


o 1 


t r 




rs 




jo t 




lo 


t 0 stor 


i t 


r 


ti 


it 




r t rs 


i t 


stor . 


t 


is 


r rst 


1 i t 




0 


t 0 i 


t r 


ti 


o i 


s ri 


fl 


s ri 


t rotot 


s st 








lo 




t 


s ri 


t 


0 str 


tio 0 s 


0 s 


s 


t 


i 




r rr 


tl 


lo i 




s 11 : 


3 s r 1 i 


ro 


ts 


i or 


or t 


i 


it. 

















1 Introduction 








V sin th Lumi oth s t in m tog phy 


t th 


n 0 


th 19th 


ntu y 1 motion pi tu s h v un gon v ious v n 


s in 


oth t 


hnology 


n ont nt 0 y motion pi tu so movi s h v st 


lish 


th ms Iv s s 


omposit t 0 m th t s V s wi ng o ultu 1 


n s 


xt n 


ing om 



n t to m ss nt t inm nt ow v onv ntion 1 movi s unlit lly p s nt 

p t min s n s n sto y s ttings so u i n s t k no p t in th m n 

m k no hoi s in sto y v lopm nt On th oth h n th us o int tion 

t hnology m k s it possi 1 o th vi w to ” om ” th m in h t in 

movi n njoy st h n xp i n li v th t this pp o h woul 

How p o u s to xplo th possi iliti so n w 1 ss o movi s om this 

vi wpoint w h V n on u ting s h on int tiv movi p o u tion 

y pplying int tion t hnology to onv ntion 1 movi m king t hniqu s 
s n initi 1 st p in ting n w typ o movi w h v p o u p ototyp 
syst m2 s on this syst m w u ntly v loping s on p ototyp 
syst m with m ny imp ov m nts his p p i fly s i s th st p ototyp 
syst m n outlin s its p o 1 m s n qui imp ov m nts h p p 

Iso int o u s th on gu tion o th s on p ototyp syst m whi h is now 

ing n y in o po ting th s i imp ov m nts 



J.-C. Heudin (Ed.): Virtual Worlds 98, LNAI 1434, pp. 107-116, 1998. 
@ Springer- Verlag Berlin Heidelberg 1998 




108 



R yohei Nakatsu et al. 



2 Outline of Interactive Movies 

2.1 Concept 

Compared with existing media, interactive movies can be regarded as audience- 
participation, experience-simulating movies. An interactive movie consists of the 
following elements: 

1. An interactive story that develops differently depending on the interaction 
of the audience; 

2. An audience that becomes the main character and experiences the world 
created by the interactive story; 

3. Characters who interact with the main character (audience) in the story. 

2.2 Configuration of the First Prototype System 

Based on the concept described above, we developed our first prototype sys- 
tem[2]. The following is a brief outline of this system. 

(1) Software 

Figure 1(a) shows the software configuration of the system. The interactive 
story consists of a collection of various scenes and a state-transition network 
between the scenes. 
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Fig. 1. Configuration of first prototype system: (a) software and (b) hardware. 

The script manager stores the data of the state-transition networks and con- 
trols the scene transition according to the interaction result. The scene manager 
contains descriptive data of individual scenes and generates each scene by refer- 
ring to the descriptive data of the scene specified by the script manager. The 
interaction manager is under the control of the script manager and scene man- 
ager, and it manages the interaction in each scene. The handlers are controlled 







t r ti Mo i irt 1 orl it rr ti s 19 
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(1) System structure concept 
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(4) Interaction manager 
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(1) Image output subsystem 
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g n ting omput g phi s t high sp us to output im g s h 

Onyx wo kst tion is us to un th s ipt mng s n mng int 

tion mng n 11 im g output so tw h t im g s p sto 

on th wo kst tions in th o m o omput g phi nim tion t in o to 

g n t omput g phi s in 1 tim kg oun omput g phi im g s 
Iso sto s igit 1 t so kg oun im g s n g n t in 1 
tim om kg oun im g s photog phi imgso Isny sto 
on n xt n 1 1 s is h multipl h t omput g phi s kg oun 

omput g phi s n kg oun photog phi im g s p o ss simult 

n ously th ough vi o o s on oth th Onyx n n igo 2 wo kst tions 
omput g phi s ispl y in 3 o mo listi im g s n 
u V s n is us to nv lop th pi y with im g s n imm s him o h 

in th int tiv movi wo 1 mg tothltn ight y p viously 

t on th wo kst tions int g t y st os opi vision ont ol n 

p oj t onto u V s n y two p oj to s On th n igo 2 n how v 

im g s output onnoinylgs n ispl y without st os opi vision 
us o p o ssing sp 

(2) Voice and emotion recognition subsystem 

oi n motion ogniz with two wo kst tions ( un 20s) th t 

Iso un th voi n motion ognition h n 1 s oi input vi mi ophon 

is onv t om n log to igit 1 y th soun o uilt into th un wo kst 
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4 Example of Interactive Story Production 

4.1 An Interactive Story 

h V p o u n int tiv sto y s on th p viously s i on 
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(a) “Romeo” controls his avatar 



(b) “Romeo” tries to touch object 
in Japanese curiosity shop 



Fig. 5. X mpl s o int tion tw n p ti ip nts n syst m 
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th s imp ov m nts in lly w illust t th op tion o on on yst m 

with th X mpl o on int tiv p o u tion o ” om o n uli t” 

References 



1. 




r ” i 




r olo i 


s 


i os” 


o o 


It 


rl 




r (196 


)• 




0 i 


k ts 




oko os 




0 r t 




liz 


tio 


0 t r 


ti 0 


i s 




t r 


o i 


tio 


t r 


o 


t 


st 




” ro is 


o t 


t r 




tio 


1 0 r 


o 


M Iti i 




o ti 






st 


s’97 


.71 77 ( 






1997). 
























3. 


tti 


M s ” o 


to 


0 t ri t t 


i 


” o 


tio 




i 


oil 


0.3 


91 




3 3 (19 9). 






















4. 


ts 


i iz t 


1. 


” o t o 


s 


i lo 








o itio 


si ross 




or 


o t t o 


str 


i or 


r 


” ro 


i 


<; 


> o 


’96 ol. 1 






14 14 


( ril 1996). 






















oko 


os 


o 


i k ts ” 


i 


lik o 




i 


tio 


t 


otio 


S 




i 


r t r ’M 




li 


ssio r 


t r ^ 


'M 




” ro 


i so 


t 




t r 


tio 1 o 


r 


0 M Iti 




i 0 


ti 






st s 


.1 19 ( 





1996). 





Real-Image-Based Virtual Studio 



ong 1 k n iki nou 

ATR Media Integration & Communications Research Laboratories 
2-2, Hikaridai, Seika-cho, Soraku-gun, Kyoto, 619-0288, Japan 
{pj i , sinoue}@mic . atr . co . jp 
http : //www.mic . atr .co.jp/ 



Abstract. In this paper, we propose a real-image-based virtual studio 
system where a virtual environment is generated by an image-based ren- 
dering technique and then a studio scene is put in the environment based 
on a 3D compositing technique. The system is implemented and the va- 
lidity of the system is confirmed through experiments. 
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3 Real-Image-Based Virtual Studio 



3.1 3D Virtual Environments 
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3.3 3D Video Composition 
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Fig. 3. xp 
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Abstract. This paper discusses a class of multimedia displays that we call pop- 
out videos. A pop-out video is a composition of multiple elementaiy displays. 
Some of these base elements are video streams, and some ai'e three-dimensional 
graphic displays. The elementary displays of a pop-out video may be related in 
two ways. They may be contiguous portions of a single virtual world, forming 
what we call an integrated video space. Alternatively, they may be parts of 
distinct spaces, forming what we call a complementary video space. In both 
cases, the elementary displays are related to each other by coordinated event 
handling. In our on-going research, we are using pop-out videos to create 
multimedia virtual worlds and to orchestrate the presentation of data in multiple 
media. After a survey of related work, the paper outlines our initial 
implementations of pop-out videos and presents future plans. 



1 Introduction 

We are building pop-out videos. These are multimedia displays that combine video 
streams and three-dimensional graphic displays. The elementary displays of a pop-out 
video may be related in two ways. They may be contiguous portions of a single 
virtual world, forming what we call an integrated video space. Alternatively, they 
may be parts of distinct spaces, forming what we call a complementary video space. 
In both cases, the elementary displays are related to each other by coordinated event 
handling. 




Fig. 1. Integrated Video Space Fig. 2. Complementary Video Space 
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Figure 1 shows an integrated video space created by Peloton [1] — a sports simulator 
that creates virtual courses for bicycle rides. The central portion of this view is a 
video stream, which is displayed as a texture on a large rectangle — a two-dimensional 
video “screen.” In the surrounding display, three-dimensional graphic elements 
represent the road and some roadside objects. The bicyclist avatars represent multiple 
simulation participants. As they explore this virtual world, users move through the 
three-dimensional synthetic terrain and view photo-realistic scenery. 

Figure 2 illustrates an example of a complementary video space. In this case, the 
video display and the graphical display are not composed into a single virtual world. 
Rather, these displays are related only by their integrated responses to system events. 
This golf instruction program helps students associate the techniques of good golfers 
with the motions of simpler graphical figures. Students watch video clips showing 
golfers — who have exemplary technique — swing clubs to hit balls; they also watch 
three-dimensional humanoid figures demonstrate corresponding animations. By 
viewing both presentations, students learn to monitor the most important elements of 
successful club swings. 

In our current research, we are creating new techniques to build pop-out videos and to 
orchestrate the presentation of data in multiple media. We are learning more about 
how to design and assemble collections of multimedia displays, and we are 
investigating ways to use them. 



2 Background 

A variety of applications integrate three-dimensional graphics with still images or 
video clips. Photographs are commonly used as textures and background elements in 
three-dimensional virtual worlds. In addition, image based modeling techniques, e.g., 
[2] and [3], permit creation of three-dimensional models from two-dimensional 
photographs. However, these applications do not represent three-dimensional spaces 
with video, and they do not permit objects to move between graphical regions and 
video regions in response to user input/output events. 

Virtual sets, e.g., [4], [5], also combine video and three-dimensional graphics. In [6], 
live actors can move within computer-generated settings. Augmented reality systems, 
e.g., [7], demonstrate another combination of these media; they lay computer- 
generated graphics over video inputs. The Interspace system [8], which supports 
multiparty conferences on the Internet, creates virtual spaces In which avatars have 
live video streams as “heads.” [9] discusses virtual reality systems, containing video 
displays, in which the relative importance of model elements (objects) is specified 
and used as a quality-of-service control for video transmission. These examples 
represent steps towards integrated video spaces; however, they do not use video for 
the representation of spaces within three-dimensional graphic models. Television and 
Web documents can be combined with WebTV devices [10]. However, these devices 
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do not create complementary video spaces because they do not provide any 
association between the television programming and the Web based materials. 

The MPEG-4 standard proposal [11] is expected to allow specification of video 
graphical hybrid environments. Its video based objects might form an appropriate 
infrastructure for pop-out videos. 



3 Integrated Video Spaces 

An integrated video space represents some parts of a virtual world as three- 
dimensional objects and other parts of the same world as still images or video. 
Because these displays are parts of the same virtual world, we call them regions of the 
world. The size, shape, and position of each region is calibrated with its neighboring 
regions to create visual unity of the composite space. A common set of controls and 
integrated event handling mechanisms are associated with the single world. Objects 
can move from one part of an integrated video space to another. Hence, we must 
handle the movement of objects between graphical and video regions. We have 
developed two techniques to deal with these inter-regional object movements. 

In the first technique, when an object moves from one region to another, the medium 
used to represent that object correspondingly changes. The new medium matches the 
one used to represent the other objects in the new region. For example, when an 
object goes from a three-dimensional region into a video-based one, it becomes a 
video element. We call this transform a merge-in because, in a sense, the object has 
“merged into” the video panel. 

Figure 1 illustrates a world in which an avatar has moved from a graphical region into 
a video region and has become a video element. Figure 3 is a behind-the-scenes view 
of the same merge-in. On the far left, a semi-transparent avatar represents the red 
cyclist’s “real” position in the virtual world. In conventional three-dimensional 
worlds, the video panel would occlude the yellow cyclist’s view of the red cyclist’s 
position. However, by performing the merge-in, Peloton allows the red cyclist to 
remain visible to the yellow cyclist. Using the red cyclist’s real position, the 
animation system translates and scales the red cyclist’s video panel for the yellow 
cyclist’s view. Figures 5 shows a close-up from this point of view. 

In our second technique, we permit objects to be designated as traceable objects. 
When a traceable object moves from a three-dimensional region to video panel, it 
does not become a video element. Instead, it is replaced in the three-dimensional 
foreground by a trace object. A trace object is an element of the foreground that 
represents an object “behind” the video panel. Figure 4 shows the same scene as 
Figure 3, but this time a trace object is used to maintain the red cyclist’s visibility. 
The animation system translates, scales, and maintains orientation of the trace object 
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Fig. 3. Behind-the-Scenes View of a Merge-In 




Fig. 4. Behind-the-Scenes View of a Trace 




Fig. 5. Yellow’s View of a Merge-In Fig. 6. Yellow’s View of a Trace 



for the yellow cyclist’s view, according to the red cyclist’s real position. From the 
yellow cyclist’s viewpoint, it is not possible to distinguish between the “real avatar” 
and the trace object. Figure 6 shows a close-up of the yellow cyclist’s view of the 



scene. 
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4 Complementary Video Spaces 

In this type of pop-out video, the video display does not integrate with other displays 
to form a single virtual world. Rather, the displays are related only through their 
coordinated response to the events that their common application handles. Without 
visual unity of a composite space, the common control interface and integrated event 
handling are created as distinct system components. Without regions for a common 
space, objects do not seamlessly move from one display to another. However, the 
elements represented in these regions are related through system actions. 

In Figure 2, both regions display the same golfer but in different media. On the left- 
hand side, the golfer is represented as video, whereas on the right-hand side, it is 
represented as a three-dimensional humanoid. This configuration allows users to 
manipulate elements, i.e. the golfer, in one region and to see the effects in both 
regions. Multiple representations of data is not a new concept. What is new in 
complementary video spaces is the ability to associate manipulations in a graphical 
region to the ones in a video region (and vice-versa). For example, playing back the 
video stream of the golfer may trigger the animation of the humanoid. When the two 
regions respond “in synch,” users experience the photo-realistic view of the video and 
also see the humanoid’s swing from several viewpoints. Handling of events can also 
result in asynchronous responses. 

To implement the environment described above, complementary video systems must 
provide users with a set of controls to perform the manipulations. Also, these systems 
must be able to “translate” the manipulations performed in one region to the 
corresponding ones in the other region. 



5 Future Work and Conclusion 

Pop-out videos provide apparent benefits in building virtual worlds as well as 
displaying information in multiple media. Shopping applications, similar to the golf 
program described above, could allow shoppers to pick objects from a video catalog 
or a commercial video and see them in other forms, or pick them from other regions 
and see them displayed in a video. Also, media melding techniques can be used 
within a multimedia virtual world to implement multimedia level-of-detail. For 
example, a surveillance application could display a region near the viewer (camera) in 
video, while displaying more distant areas through other media. Pop-out videos could 
enhance television programming by coupling video with information represented in 
other media. The closed caption (subtitle) concept of today’s television allows text to 
be associated with video content. This concept can be extended to associate other 
information — displayed in other media — with video content. For example, hyper- 
textual Web documents and three-dimensional virtual worlds could be associated with 
video programming content. Various suppliers, including the producers of the 
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original programming content or specialized providers, could supply these additional 
materials. In fact, several collections of additional material could be available for 
each program. Consumers could then select their preferred collections. 

Pop-out videos can also be used to build distributed virtual worlds that react to 
performance constraints. Since region boundaries within a virtual world can vary 
dynamically, the relative media mix of a world can vary correspondingly. This 
variation provides an opportunity to tailor applications and their presentation of 
information to the needs and resources of the system user. For example, it could 
improve the performance of a distributed application executing on heterogeneous 
computers connected via the Internet. Application users with powerful machines but 
poor network connections could take advantage of their computing power by 
specifying that a virtual world be presented mainly as three-dimensional objects 
rendered locally. On the other hand, users with network computers — offering high- 
end network connections but less processing power — could specify that the scene be 
presented using minimal three-dimensional representation to view most of the scene 
as streamed video. 

Although many issues still need to be resolved, these examples show that the 
integration of video in richer multimedia environments could open the doors of a very 
large set of interactive multimedia applications. 
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Abstract. A method was developed to segment an image foreground 
and background based on color content. This work presents an alter- 
native to the standard blue-screen technique (weather man method) by 
exploiting the color shifts of light sources and filtering each camera lens 
and correcting white balances for each camera, we compressed the col- 
ors of the scene background into a chromaticity subspace to make the 
foreground-background segmentation easier to perform. The segmenta- 
tion is singular decomposition (SVD) based. 



Keywords: olor gm nt tion, lu r n, 3 ooting, irtu 1 lity, 

rom -k y 

1 Introduction 

ppli tions d m nding n utom ti vid o str m s gm nt tion r num rous 
n oft most f mous ist lusr nw t rmn)mtodw r suj t 

situ t d in front of uniform lu w 11 or y loid, is ov rl id on n ot r im g 
y n utom ti rom -k y s gm nt tion n t 1 vision produ tions, t lu 

s r n is us d in studio nvironm nt nd Hows limit d possi 1 positions nd 

ori nt tions for t m r s n virtu 1 lity ppli tions su s t 
1 , on w nts to s gm nt for ground su j t from kground s n using 
t p rsp tiv of n r itr rily position d nd ori nt t d m r , uilding 360 
d gr s lu s r n is not usu lly f si 1 , s it would imply to p int n ntir 
sp r in lu , position t su j t insid nd provid ol s for m r 1 ns 

ddition lly, t lusrnmtodr quir s p rti ul r tt ntion for s dows 

proj t d on t lu w 11 nd t qu lity of t s gm nt tion d p nds on t 

uniformity of t lig t spr d on t kground 

n our ppro , t lig ting of t kground is mor tol r nt T is 1 im is 
justi d us w do not v d rk surf s w i g n r t nois ur m t od 
is simpl to r li in pr ti , nd do sn’t r quir ution on kground lig t- 

ing in t rms of uniformity nd s d s of illumin tion not r import nt im- 

prov m nt is t t som on n w r lu lot s nd still s gm nt d orr tly 
n d pt t s gm nt tion to t su j t nd t kground s n inst d 
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of imposing onstr ints on t tors lot s nd kground olors T is pro- 
vid s n w fl xi ility s ow t t our m t od ould xt nd d to st nd rd 

outdoor s ooting wit f w onstr ints w i ould m k it v ry us ful 

T is work provid s t r import nt n ts t 1) provid s not r solu- 
tion to t for ground- kground s gm nt tion pro 1 m y using lo 1 lig ting 
onditions to for s p r tion of olors in t rom ti ity olor su sp ; it 
2) dis uss s t us of t w it In nd xploits t r 1 tions ip tw n 
w it In s ift nd t olor tint; nd 3) n lly provid s solution for or- 

r ting wrong m r w it In quir d in t d t tion 2 dis uss s 

t t ory nd d V lopm nt of t on pts us d to support t is work tion 

3 ov rs t olor orr tion t niqu s tion 4 dis uss s n s d s g- 

m nt tion nd olor lust ring m t od tion 5 d s ri s t xp rim nts w 

ondu t d nd provid s dis ussion nd on lusion of t r suits 

2 Theory and Concept Development 

nd rst nding t r suits w pr s nt r r quir s 1 r und rst nding of r - 
li ing t olor s ift t roug It ring nd s ifting oft m r w it In 
n ddition, t p ysi s of t lig t sour s, t ppropri t It rs, nd t ir 
imp t on t olor ompr ssion in t rom ti ity olor su -sp must 

und rstood in lly 11 of t im g d t is quir d y m r , t r for , t 
pit-f 11s of d t quisition using ” off-t -s If’ nd ” ro d st qu lity” m r s 

s ould Iso xplor d 

2.1 The Color Shift 

nst d of s ooting su j t in front of sp i olor d kground, w ng 

t sp trum of t for ground lig t y pplying olor d It r in front of t 

ul f w orr t t is s ift only for t for ground nd not for t kground 

w ” ompr ss” t ntir kground in r gion of t olor sp T r r 

two w ys of doing t is 

— Tuning t w it In of t m r s) on t t mp r tur of olor of 
11 t ul s involv d in t s ooting usu lly 3200 K for indoor lig ts, w 
us d 650 ) dding It r on t 1 ns of t m r , t is It r is n 

opposit olor of t on position d on t for ground lig ts did two 
t sts in using som os o It rs 

1 T st 1 ull lu #3202 oosts 3200 K to 5500K) on for ground lig ts 

nd os osun85 #3401 onv rts 5500K to 3200K) on t m r 1 ns 

2 T st2 If lu #3204K oosts 3200 K to 4100K) on for ground lig ts 

nd os osun If T #3408 onv rts 5500K to 3800K ) on t 

m r 1 ns 

nts xp rim nts, w us d Imost ompl m nt ry It rs , t n us 

t is ompl tion is f r from p rf t t r suit of t orr tion will littl 
it tint d or ng not d sir d s ift t t w ould orr t y t olor 
orr tion m t od xpl in d furt r 
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— s ond m t od for doing t is, is to djust t m r w it In to 
t lig t spr d y t for ground sour s ving t If In or full In 
It r on) T is m t od is us full only for orr ting sm 11 s ifts ow v r, it 
giv s som tt r r suits 



2.2 Light Sources 

T It rs w r not t uniqu olor s ifts tu lly t w it of qu 1 n rgy 

do sn’t xit, 11 t w it lig ts w ns r in f t mor or 1 ss tint d n 

pproxim tion of t sp trum of t w it lig ts ul s is giv n y t In k’s 
qu tion 

) 1 = -l)-i -3 1) 

r 1 3 7415 10-16 ^ nd 2 1 4388 10-2 m K T is t t mp r tur 

of olor of t sour t olor of t sour is usu lly d s ri d y t quiv 1 nt 

1 n ki n r di tor n our xp rim nts, t ul s d t mp r tur of olor of 

3200K indoor lig ts) pplying som olor d It rs to sour is quiv 1 nt to 
produ t of t lig t sp trum nd t sp tr 1 tr nsmission f tur s of t 

It r igur 1 s ows t sp trum of t lig t spr d on t for ground v r 1 



= 1 



Fig. 1. suiting lig t sp trum from If lu It r dd d on 3200K w it 
lig t; )sp trum oft 3200Kw it lig t f ); ) tr nsmission f tur soft 

If lu It r g ); nd ) t r suiting for ground lig t sp trum f )g ) 





sptrmyrtt sm stimulus, t is prin ipl is 11 d ”m t m rism” 
T is p nom n is wid ly us d in vid o to synt si sp i olor from only 

3 r ys of olor d lig ts or x mpl onsid r t r d, gr n nd lu prim ri s 

frn s23564 giv s good sis to st rt wit 

not r r pr s nt tion of lig t sour p ssing t roug It r would 
mixtur of only on r y of pur olor nd w it of quiv 1 nt n rgy r 

t tint is giv n y t w v 1 ngt of t r y, t s tur tion is giv n y t 

p r nt g of w it dd d to t is olor, nd t int nsity is giv n y t tot 1 

n rgy of t is mixtur n ord r to o t in t is n w r pr s nt tion w i is si r 

to int rpr t, w n d to omput produ t of onvolution of t sp trum of 

stimulus nd t t r urv s of t s nsitivity of st nd rd um n vision 
syst mT sdnd stoftr urv s ~ ~g )or ) 789 

giving t s nsitivity of st nd rd ” y ” to sp i olor T us, w o t in 
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/ ) )- ) 

f ) r ) 

f ) r ) 2 ) 

100 

/ ( n )d 

r is t illumin tion sour , is t sp tr 1 r fl t n f tor, nd 
, nd , r t tristimulus v lu s in t r pr s nt tion n now 

r pr s nt t stimulus in t rom ti ity di gr m x,y) n t x mpl 

low, w r studying ow s ift t lig t sour , t n w ’ll onsid r t t on 
t ntir visi 1 sp trum 

+ + + + 1 — — 3) 

T igur 2 s ows ow to g t t tint of stimulus , y omputing t 



Table 1. x,y oordin t s in t 
It rs 



rom ti ity su sp of t lig ts nd 



Light 3200K 


Light 3200K with 
a Half blue filter 


Light 3200K with 
a Full blue filter 


X 0.4234 
y 0.3991 


0.3808 

0.3806 


0.3387 

0.3316 



int rs tion of t lin joining t w it of qu 1 n rgy ) nd t studi d 
olor ) nd t ound ry of t urv s or t purpl olors, w i do 




Fig. 2. nt rpr t tion of s ift d w it in t di gr m 



not orr spond to uniqu r y of lig t, w will us — for d s ri ing t tint 
t is is tu lly t tint of t opposit olor) in t purity of x it tion s 
som int r sting g om tri prop rti s, w ’ll us it 1 t r for t olor orr tion, 
ut t olorim tri purity is mor pr is , t is is t on will us r 
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r is t lumin n of t w it nd t lumin n of t pur 
olor on t ound ry of t di gr m or dding two stimuli w know Iso 
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w r only 


onsid ring t 


lig t sour t 
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tion is 100 


on t 



ntir sp trum , t r suiting stimulus ) is giv n y w will only do t 
proof for , sin nd r simil r) 



1- ) 6) 

T is int rpr t tion r pi s ny olor d stimulus y n ddition of n qu 1 n- 
rgy w it nd pur olor r y T s r 1 tions ips r v ry us ful for orr ting 
olor s ift 

ur go 1 is to s gm nt utom ti lly su j t situ t d in t for ground 

of s n f w lig t t for ground wit lu lig t nd t kground 

wit 3200K lig t, t kground r quir s lig ting wit mor ”or ng lig t” 

t n t for ground T is ff t will s ift t olor of t kground tow rd t 

or ng y llowis r gion of t rom ti ity su sp f w orr 1 1 for ground 
lig t t t m r t r suit will norm 1 for ground w il t kground 

g ts ompr ss d into si t d r gion of t rom ti ity su sp 

T rst w y of doing so is to tun t w it In of t m r on 
3200K nd Iming t roug n or ng It r ompl m nt ry to t lu It r 

on t for ground lig ts f t It rs r r lly ompl m nt ry, t for ground 

lig t is p r iv d w it 3200K) nd t ntir kground is s ift d in or ng , 
it full lu It rs on t for ground lig ts nd full or ng It r on t 
m r 1 ns, t kground is s ift d of full or ng w il t for ground 

pp rs orr tly wit Ind nft, in our xp rim nts t is orr tion 

is not p rf t us t It rs w r not ” x tly” ompl m nt ry T r r 
no p rf t ompl m nt ry It rs omm r i lly v il 1 t t is tim N xt w 

pr s nt w y of orr ting s ift d im g ft r ing s gm nt d 

n mig t int r st d in s ifting t kground y mor t n on full 

or ng in ord r to p rform tt r s gm nt tion s d on olor t is possi 1 to 

dd som full or ng It rs on t kground lig ts n t is s t kground 

will s ift d twi of full or ng w i m k s t kground ompl t ly 

ompr ss d in t or ng su sp of t r pr s nt tion 

ur m t od impli s mor pow r to lig t n t for ground t n in lu 
s r n m t od, for inst n , f t lig t sour is n qu 1 n rgy lig t, nd t 

It rs ppli d on t lig t sour nd 1 ns r r sp tiv ly full lu nd full 
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or ng It r, t m r wills only 10 to 30 oft n rgy of t lig t sour 
ow V r id Is It rs sp trum of t orr tion fl t for v ry ) ould m k 
t m r s 50 of t sour n rgy us d not r w y for orr ting 

t s ift loosing 1 ss n rgy, using w it In orr tion inst d of full 

or ng It r on t 1 ns s 

om V ry int r sting f tur s r positioning t is m t od s n It rn tiv 
to t popul rlusrnmtod st rdr s pro ly noti d, w don’t 
n d to V p rti ul r kground T st s n rio would to lig t n t 
kground wit its most ompl m nt ry olor T is m t od do sn’t d m nd 
p rf tly uniform lig t on lu w 11 Lig t ning it will noug Lik in 

t rom -k y pro ss, t Ik surf s will diffi ult p rt to pro ss 

us 1 k is V ry s nsitiv to nois in olor n our post pro ssing orr tion 
w r mov t Ik und r t r s old nd r pi t my int rpol tion using 

t n ig ors of t r mov d r 

T is m t od n Iso work for outdoor s n s n p otogr p rs r 
usu lly using gold n r fl tors to simul t suns t, t y r in f t s ifting t 
for ground wit lig t tint d diff r ntly t t t kground orr tion on 

t 1 ns is don y t ompl m nt ry It r to t r fl tor 



2.3 The Cameras 
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to do t w it In or inst n , m r is not 1 to orr t s ift don 
y or ng It r on t 3200K lig t, ut would p rf tly 1 to orr t 
s ift don y n or ng It r on lig ts 

mixtur of t two m t ods mig t wort to us 

3 Color Correction 

propos t following m t od to p rform orr tion in ord r to m lior t 
n im g , or in ord r to r tri v olor s ift du to non- ompl m nt ry It rs in 

t rst m t od T is olor orr tion m y Iso int r st t os w o r on rn 

y vid o s qu n quir d wit poor w it In d nd w nt to r tri v 
mor pt 1 r suit 

will try to simul t t r 1 stimuli, s d on t d t w quir d T 

rst st p is to nd t r 1 w it of t im g w r n ly ing T is m y 

giv n m nu lly y indi ting w it of r f r n , or utom ti lly if on s 

s ot olor rs, w i is t s in our xp rim nts us d 75 olor 

r rt wit 100 w it r f r n , in ord r to r t mod 1 for our m r 

for som p rti ul r s ifts rst s ot t rt und r t 3200K lig ts nd 

tun d t w it In to g t t st w it s possi 1 on t v tor s op 
T is rt is t n our r f r n for t visi 1 sp trum n furt r work w will 
pro ly d sign our own rt in luding mor olors w 1 of 300 li r t d 

olors) ft r s ooting t is rt on und r t r f r n lig t, nd s ond 

tim und r t s ift d lig t, w r 1 to r t ro ust mod 1 of t m r 

us d nd for p rti ul r s ift ur urr nt mod 1 is only using 6 olors for 
int rpol ting t ntir sp trum, ut t orr tion o t in d is Ir dy quit 
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Abstract. A methodology is described for designing real-time animated artifi- 
cial life worlds consisting of populations of physically-based articulated crea- 
tures which evolve locomotion anatomy and motion over time. In this scheme, 
increasing levels of emergent behavior are enabled by designing more auton- 
omy into the simulation on progressively deeper levels. A set of simulations are 
discussed to illustrate how this methodology was developed, with the most re- 
cent simulation demonstrating the effects of mate choice on the evolution of 
anatomies and motions in swimming creatures. 



1 Introduction 

The art of animation as invented by the Disney animators has been called "The Illu- 
sion of Life." It seems that Computer Animation as a craft has never really caught up 
to the expressivity, humor, and lifelike motion possible with classic cel-based char- 
acter animation. The animators at Pixar have been successful at adapting computer 
technology to the fine art of classic character animation technique. But as feature film 
animators refine this marriage of new and old technologies, a new form of animated 
character emerges, not in the film industry but from within various artificial life labs. 
Add to Disney's "Illusion of Life," the "Simulation of Life," and witness a new tech- 
nology-one which is compatible with the goals of Virtual Reality-a future cyber- 
space in which characters are not just animated: they are autonomous, reactive agents 
as well. 

This paper describes an approach to artificial life (alife) [9], stemming from ani- 
mated art and a design process for creating it. It is an exploration of autonomously 
generated motion and form. The original impetus is not biological science, although 
biology has become an important aspect of the work. The angle on alife discussed in 
the paper may provide ideas and techniques that are useful in creation of populated 
virtual worlds for entertainment, education, and research. 

I will discuss a family of simulations which were built upon each other, in which 
populations of artificial creatures evolve anatomy and motion over time. In these 
simulations, emergent behavior is enabled by progressively designing levels of 
autonomy into the model. The simulations are used to illustrate how this methodology 
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was built, with the most recent simulation demonstrating the effects of mate selection 
on the evolution of anatomies and motions in swimming creatures. 

Future enhancements will include wiring up an evolvahle nervous system con- 
nected to various sensors and actuators, within which neural net-like structures might 
emerge, potentially enabling the creatures to evolve higher-level stimulus-response 
mechanisms and "states of mind," which emerge through direct contact with the envi- 
ronment. 

Design Considerations. A main goal of alife projects is to construct models in which 
self-organization and adaptive behaviors can arise spontaneously, not hy design, but 
through emergence. A duality is observed in the creation of real-time alife worlds: 
while the goal of an alife simulation is emergence, it is ultimately a designed artifact. 
Creating alife worlds, in this methodology, is a matter of designing emergence. 

One must choose a level of abstraction, below which a number of assumptions are 
made about the ontology supporting the model. Above this level of modeling, emer- 
gent phenomena arise as a product of the design of the subsystem when the model is 
run. For instance, many alife simulations model populations of organisms whose 
spatial positions are represented by locations in a cellular grid, and whose physical 
means of locomotion (jumping from cell to cell) are not clearly specified. The emer- 
gent properties in question may not require a deeper level of simulation than this. 
Flowever, a simulation in which an articulated body plan is integral to locomotion (as 
well as reproduction) requires a lower level of abstraction, and a deeper physical 
model becomes necessary. More design may be required to build the simulation 
foundations, however, it may allow for more complex emergent phenomena. 

Towards the quest for more autonomy, the most recent simulation in the set which 
I will discuss consists of a population of many hundreds of swimming creatures 
which animate in real-time. The creatures come in a large variety of anatomies and 
motion styles. They are able to mate with each other, and choose who has the "sexi- 
est" motions, enabling evolution of swimming locomotion and anatomy which is 
attractive (beauty of course being in the eye of the beholder). The best swimmers 
reproduce their genes (because they can swim to a mate), and the most "attractive" 
swimmers get chosen as mates. In this simulation, not only are the aesthetics of mo- 
tion and form subject to the chaotic nature of genetic evolution, but the creatures 
themselves partake in the choice of what forms and motions emerge. 

The methodology discussed in this paper includes the following key components: 

• a morphological scheme with an embryology 

• a motor control scheme 

• a physics 

• a nervous system 

• genetic evolution 

• computer graphics rendering 

Each of these components are described throughout the sections below. They are not 
explained in great technical depth, but rather discussed as a general methodology, 
with some commentary. 
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1.1 Related Work 

Early examples of using alife principles in computer animation include Boids [12], in 
which complex life-like behaviors emerge from many interacting agents. Badler [2] 
describes physically-based modeling techniques and virtual motor control systems 
inspired by real animals used to automate many of the subtle, hard-to-design nuances 
of animal motion. In task-level animation [22] and the space-time constraints para- 
digm [21] these techniques allow an animator to direct a character on a higher level. 

The genetic algorithm [7, 6] has been used for the evolution of goal-directed mo- 
tion in physically-based animated figures, including a technique for evolving stimu- 
lus-response mechanisms for locomotion [10]. Sims [14, 15] has developed impres- 
sive techniques for the evolution of morphology and locomotion using the genetic 
programming paradigm [8]. Also, a holistic model of fish locomotion with percep- 
tion, learning, and group behaviors, which generates beautifully realistic animations, 
was developed by Terzopoulos, Tu, and Grzeszczuk [16]. 



1.2 Walkers 

The first project in the series began an attempt to build a 3D stick figure which could 
stand up by way of internal forces bending various joints. The original goal was to 
model a human-like form, but this was too large a task at the time. To simplify the 
problem, the bipedal figure was reduced to the simplest anatomy possible which can 
represent bipedal locomotion: two jointed sticks, called "Walker," shown in Figure 
la. 

Performance artists Laurie Anderson once described walking as a continual process 
of catching oneself while almost falling, by placing one foot out in the direction of 
the fall, stopping the fall, and then almost-falling again [1]. This notion inspired the 
idea to construct a 3D bipedal figure which has mass and responds to gravity and 
which is always on the verge of falling over and so must continually move its legs in 
order to keep from falling, to stay upright, and to walk. 

Walker's "head" experiences a pull in the direction of its goal (an inaccurate model 
of the desire to be somewhere else, yet sufficient for this exploration). Walker can 
perceive how much its body is tilting. The amount of tilt is used to modulate the mo- 
tions in its legs. As it's head is pulled towards its goal, its legs move in such a way as 
to keep it upright and afford locomotion. Internal forces cause the legs to rotate about 
the head joint, back and forth, in periodic fashion, using sine functions. Parameters in 
the sine functions for each leg, such as amplitudes, frequencies, and phases, are ge- 
netically determined, varying among a population of Walkers, and are allowed to 
evolve over time within the population. Responses to body tilt are also genetically 
determined. 

At the start, the motions of most of the Walkers are awkward and useless, with 
much aimless kicking, and so most of the Walkers immediately fall as soon as they 
are animated, although some are slightly better at staying upright for longer. By using 
a simplified genetic algorithm which repeatedly reproduces the best Walkers in the 
population, with chance mutations, and killing off the rest, the population improves 
locomotion skill over time. 

To demonstrate Walker's ability to react to a constantly changing situation, a virtual 
"leash" was attached to the head, which a user could gently pull around. Also, see- 
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narios were constructed in which multiple Walkers had attractions and repulsions to 
each other, generating novel pursuit and evasion behaviors. 

Walker is the first in a series of alife creatures in this system consisting of linked- 
body parts which are connected at their ends and which can rotate about each other. 
Figure lb shows a creature from a later simulation of 2D creatures consisting of five 
interconnected segments, with four effective joints. As in the case of Walker, rota 
tions about the joints are controlled by sine functions with evolvable parameters. 
Fitness in this scheme is based on distance traveled. Although creatures in this popu- 
lation are 2D, locomotion is possible, with evolved populations consisting of crea- 
tures ambulating either to the left or to the right. 




Fig 1. (a) Walker, (b) 2D walking figure. 



2 A Morphology Scheme 

It is one thing to link together a set of sticks in a predetermined topology and then 
allow the motions of the joints to evolve for the sake of locomotion. It is another 
thing entirely to allow morphology itself to evolve. 

In the next incarnation of this alife methodology, a morphological scheme was 
designed which allowed the number of sticks, lengths, branching angles, and stick 
topology to vary. All of these factors are genetically determined. An important prin- 
ciple in this newer morphological scheme is that in this case, there are no implied 
heads, legs, arms, or torsos, as shown in Figure 3. Creatures are open-ended structures 
to which Darwinian evolution may impart differentiation and afford implicit function 
to the various parts. 





Fig. 3. 3D creatures with variable morphology. 
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An extension of this scheme includes bodies with variable thickness among parts. 
Figure 4 shows a representative 3D creature from this set. These creatures are the 
most complex of all the morphological schemes, and can exhibit locomotion schemes 
and anatomies which exploit the effects of uneven distributions of mass among the 
body (consider for instance the way a giraffe uses the inertia of its neck when it runs). 
The figure in this illustration uses a tripod technique to stabilize itself. Creatures in 
this scheme tend to evolve 3 or 4-legged locomotion. Bipedal locomotion does not 
evolve: this is probably due to the fact that they have no sensors for balance, and thus 
no way of adjusting to tilt (as Walker did). A future enhancement will be to add 3D 
"vestibular" sensors, to enable bipedal locomotion. 




Fig. 4. A 3D creature with variable thickness in parts. 



2.1 Embryology 

Each creature possesses a set of genes organized in a template (the genotype). Geno- 
types are represented as fixed-length strings of real numbers. Each gene is mapped to 
a real or integer value which controls some specific aspect of the construction of the 
body, the motion of body parts, or reactivity to environmental stimuli. Genotypes get 
expressed into phenotypes through this embryological scheme. The typical genotype 
for a creature includes genes for controlling the following set of phenotypic features: 

number of parts 

the topology (branching order) of parts 

the angles at which parts branch off other parts 

colors of parts 

thicknesses of parts 

lengths of parts 

frequencies of sine wave motions for all parts 
amplitudes of sine wave motions in each part 
phases among sine wave motions in each part 
amplitude modulators of each sine wave (reactivity) 
phase modulators of each sine wave (reactivity) 

The design of a genotype-phenotype mapping has been one of the most interesting 
components of the design process in building alife worlds in this scheme. It is not 
always trivial to reduce a variable body plan to an elegant, simple (and evolvable) 
representation. Dawkins [5] promotes the concept of a "constrained embryology," 
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whereby global aspects of a form (such as bilateral symmetry or segmentation) can be 
represented with a small set of control parameters. A representation with a small set 
of genes which are able to express, through their interactions during embryology, a 
very large phenotypic space, is desirable, and makes for a more "evolvable" popula- 
tion. 



2.2 Motor Control 

The motor control scheme in this system is admittedly simplistic. Real animals do not 
move like clockwork, and do not move body parts in periodic fashion all their lives. 
However, as mentioned earlier, one must choose a level of abstraction. For the simu- 
lations designed thus far, this simplistic model of animal locomotion has been suffi- 
cient. Also, the addition of modulators to the periodic motions makes the motion less 
repetitive, and enables reactivity to the environment. A more reactive motor control 
scheme would be required if any behavior other than periodic locomotion is expected 
to evolve. 



3 Physics 

This alife system evolved from a simple animation technique whereby motion was 
made progressively more complex. The notion of a physics also became incorporated 
in progressive steps. In the figures modeled in this scheme, deeper levels of physical 
simulation were added as needed, as more complex body plans were designed. 

In the present scheme, time is updated in discrete steps of unit 1 with each time 
step corresponding to one update of physical forces and one animation frame. A 
creature is modeled as a rigid body, yet which can bend its parts as in an articulated 
body. Torque between parts is not directly modeled in this scheme. The effects of 
moments of inertia and changing center of mass are modeled. A creature's body has 
position, orientation, linear velocity, and angular velocity. Position and orientation are 
updated at each time step by the body's linear and angular velocities. These are in turn 
affected by a number of different forces such as gravity, collisions of body part end- 
points with the ground (for terrestrial creatures), forces from parts stroking perpen- 
dicular to their axes in a viscous fluid (for sea creatures), and in some cases, contact 
with user stimulus (a mouse cursor which pokes into the virtual space). 



4 Nervous System 

It is possible that brains are simply local inflammations of early evolutionary nervous 
systems, resulting from the need in early animals to have state and generate internal 
models in order to negotiate a complex environment. This is an alife-friendly inter- 
pretation of brains. The design methodology employed here is to wire up the crea- 
tures with connectivity and reactivity to the environment, and then to eventually plug 
in evolvable neural nets which complexify as a property of the wiring to the environ- 
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ment. Braitenberg's Vehicles [3] supply some inspiration for this bottom-up design 
approach. 

The design philosophy used here does not consist of building a brain structure prior 
to understanding the nature of the organism's connectivity to the physical environ- 
ment. If adaptive behavior and intelligence are emergent properties of life, then it is 
more sensible to design bodies first, and to later wire up a proto-nervous system in 
which brains (might) emerge. The "Physical Grounding Hypothesis" [4], states that to 
build a system that is intelligent it is necessary to have its representations grounded in 
the physical world. 

In this system, there are not yet any brains to speak of. It includes a simple mental 
model which is (at this point) purely reactive: sensors in a creature detect aspects of 
the its relation to the environment, which can directly transition its state to another 
state, or which directly control actuators (real-time changes in the motions of body 
parts). For instance, while in "looking for mate state", a creature will react to certain 
visual stimuli, which can cause it to enter into a new state, such as "pursuing a mate". 
A future enhancement includes building an evolvable recurrent neural net which 
mediates the sensor-actuator pathways, and can grow internal structure and represen- 
tation in the context of the creature's direct coupling with the dynamic environment. 



5 Evolution 

Models of Creationism are made implicit every time a craftsperson, animator, artist, 
or engineer designs a body of work and establishes a methodology for its creation. 
Darwinian models are different than Creationist models (though not absent of a crea- 
tor). In Darwinian models, important innovation is relegated to the artifact. Invention 
happens after initial Design. Surprises often result. Genetic algorithms are Darwin 
Machines [11], designed to take advantage of lucky mutations. They are serendipity 
engines. 

The earlier alife simulations discussed use a variation of the standard genetic algo- 
rithm: hundreds of creatures are initialized with random genes, then each is animated 
for a period of time to establish its fitness ranking (in most cases, determined by how 
far it has traveled from the position at which it started). After this initial fitness rank- 
ing, a cycle begins. For each step in the cycle, two statistically fit individuals are 
chosen from the population, mated, and used to create one offspring which inherits 
genes of both parents through crossover. The offspring undergoes embryology and 
then replaces the least fit individual in the population. The offspring is then animated 
to determine its fitness. The process continues in this way for a specified number of 
cycles. This technique guarantees that overall fitness never decreases in the popula- 
tion. The most fit creatures are never lost, and the least fit creatures are always culled 
[17]. Figure 5 shows 4 snapshots of a locomotion cycle showing a 3-legged anatomy 
and locomotion strategy, which evolved from one of these simulations. 



Fig. 5. An evolved 3-legged locomotion strategy. 
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6 Sexual Swimmers 

In the latest series of simulations, dimensionality was brought down from three to 
two, but the model was deepened by allowing reproduction to be spontaneous. To 
ground the simulation more in a physical representation, the concept of a "generation" 
was blurred. Instead of updating the entire population in discrete stages (a schedule 
determined by the "creator"), this system allowed reproduction to occur asynchro- 
nously and locally among creatures able to reach proximity to each other in a spatial 
domain. The notion of an explicit fitness function is thus removed [18]. 

Locomotion behavior evolves globally by way of local matings between creatures 
with higher (implicit) fitness than others in the local area. Figures 6a and b illustrate 
two swimming styles which emerged from this simulation. 
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Fig. 6. Sequences of images of swimming strategies: (a) paddling-style, and (b) undulating- 
style, which emerged through evolution. 

Turn That Body. In order to pursue a potential mate which is swimming around 
(possibly pursuing its own potential mate), a swimming creature has to be able to 
continually orient itself in the direction it wants to go. A 
reactive system was designed which allowed the creatures to 
detect how much it would need to turn its body while swim- 
ming (analogous the the "tilt" sensor in Walker, only now 
covering 180 degrees in a plane). Essentially, the creature 
senses the angle between the direction of its goal and its own 
orientation as the stimulus to which it reacts, as shown in 
Figure 7. 

As opposed to Walker, which experiences a "magic pull" in the direction of its goal, 
these creatures must create their own means of propulsion, using friction with the 
surrounding fluid. 

Since the number of possible morphologies and motion strategies in these creatures 
is vast, it would be inappropriate to top-down-design a turning mechanism. Since 
turning in a plane for a 2D articulated body can be accomplished by modulating the 
phases and amplitudes of certain part motions, it was decided that evolution should be 
the designer, since evolution is already the designer of morphology and periodic 
motion. Thus, modulator genes (shown in the list above) were implemented which 
can affect the amplitudes and phases in each part's motions in response to the stimu- 
lus. 
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Fig 7. Turning stimulus 
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A commercial product was derived from this simulation and called "Darwin Pond" 
[13]. It enables users to observe hundreds of creatures, comprised of 2D line seg- 
ments, representing a large phenotypic space of anatomies and motions. They can be 
observed in the Pond with the aid of a virtual microscope, allowing panning across, 
zooming up close to groups of creatures, or zooming out to see the whole affair. One 
can create new creatures, kill them, move them around the Pond by dragging them, 
inquire their mental states, feed them, remove food, and tweak individual genes while 
seeing the animated results of genetic engineering in real-time. Figure 8 shows a 
screen shot of the basic Darwin Pond interface. 




Fig. 8. Darwin Pond interface. 

Physics and Genetics Are Linked. It is important to point out that in this simulation, 
genetic evolution is totally reliant on the physical model which animates the crea- 
tures. Not only does it take two to tango, but they have to be able to swim to each 
other as well. In this simulation, genotypes are housed in physically-based pheno- 
types, which, over evolutionary time, become more efficient at transporting their 
genotypes to other phenotypes, thereby enabling them to mate and reproduce their 
genes. This is the selfish gene at work. The emergence of locomotive skill becomes 
meaningful therefore in the context of the environment: it becomes physically 
grounded, and the fitness landscape becomes dynamic. 

A variation on this simulation, called "Gene 
Pool" [20], was developed, and included a larger 
genotype-phenotype scheme, variable thickness 
in body parts, a deeper physics, and a conserved 
energy model. Figure 9 illustrates a collection of 
these 2D swimming creatures (before evolution 
has had any effect of body plan). They are called 
"swimbots". 



What's Sex Got To Do With It? Gene Pool introduces mate choice in the process of 
reproduction. In this scheme, not only is the evolution of form and motion more 
physically grounded, and subject to the local conditions of the simulated world, but 
the aesthetic criteria which determine the "sexiest" motions also become a factor for 
reproduction. 




Fig. 9. Swimbot anatomies 
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What does "sexy" mean? Sexual selection in evolution is responsible for phenom- 
ena such as the magnificent peacock tail, and the elaborate colorful dances of many 
fish species, which sport curious body parts, adapted for mate attraction. Attraction- 
based features in many species may be so exaggerated and seemingly awkward that 
one would assume they decrease the overall fitness of the species, in which locomo- 
tion efficiency, foraging, and security are important. I arrived at an hypothesis: could 
mate preferences for arbitrary phenotypic features inhibit the evolution of energy- 
efficiency in a population of locomotive creatures? For instance, if the creatures in the 
population were attracted to slow-moving bodies, and only mated with the most sta- 
tionary creatures in the population, would locomotion skill emerge? If so, what kind? 
What if they were all attracted to short, compact bodies? Would the body plan be- 
come reduced to a limbless form, incapable of articulated locomotion? If they were 
attracted to wild, erratic motion, would the resulting swimbots be burning off so 
much energy that they usually die before reproducing? 

As an experiment, a simulation was built, including a variety of mate preference 
criteria determining which swimbots would be chosen [19]. In choosing a potential 
mate, a swimbot would take snapshots of each swimbot within its local view horizon, 
and then compare these snapshots and rank them by attractiveness. Criteria settings 
included: attraction to long bodies; attraction to lots of motion; attraction to bodies 
which are "splayed out," and attraction to massive bodies. The inverses of each of 
these phenotypic characteristics were also tested. As expected, in simulations in 
which length was considered attractive, populations with elongated bodies with few 
branching parts emerged. In these populations, locomotion was accomplished by a 
tiny paddle-like fin in the back of the body, or by gently undulating movements. 
Figure 10 illustrates a cluster of swimbots resulting from this simulation. 



\ S 



Fig. 10. Swimbots which evolved through mate preferences for long bodies. 

Attraction to "splayed-out" bodies (in which the sum of the distances between the 
midpoints of each part is larger than average), resulted in creatures which spent a 
large part of their periodic swimming cycles in an "open" attitude, as shown in Figure 
1 1 . In this illustration, the creature is seen swimming to the lower right. Approxi- 
mately one swim cycle is shown. Notice that the stroke recovery (the beginning and 
end of the sequence) demonstrates an exaggerated "opening up" of the body. It is 
likely that this behavior emerged because snapshots can be taken at any time by 
creatures looking for a mate, and so the more open a potential mate's body is, and the 
longer amount of time it is open, the more attractive on average it will be. 

These experiments show how mate preference can affect the evolution of a body 
plan as well as locomotion style. The effects on energy-efficiency were non-trivial: 
locomotion strategies emerged which took advantage of attraction criteria yet were 
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still efficient in terms of locomotion. As usual with many alife simulations, the phe- 
notype space has unexpected regions which are readily exploited as the population 
adapts. 
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Fig. 11. Swimbots which evolved through mate preferences for splayed bodies. 



6.1 Realtime Evolution 

The alife worlds described here incorporate physical, genetic, and motor-control 
models which are spare and scaled down as much as possible to allow for computa- 
tional speed and real-time animation, while still allowing enough emergent phenom- 
ena to make the experience interesting and informative. The effects of Darwinian 
evolution can be witnessed in less than a half-an hour (on an average Pentium com- 
puter). For educational and entertaining experiences, it is important that there be 
enough interaction, immersion, and discovery to keep a participant involved while the 
primordial soup brews. The average short-attention- span hardcore gamer may not 
appreciate the meditative pace at which an evolutionary alife system runs. But alife 
enthusiasts and artistically or scientifically oriented observers may find it just fine. 
For watching evolutionary phenomena, it sure beats waiting around a couple million 
years to watch interesting changes. 



7 Computer Graphics Rendering 

For the 3D stick figures, thick, dark 3D lines are drawn to represent the sticks. The 
interesting (and important) part comes in drawing a shadow, since this is largely the 
graphical element which gives the viewer a sense of the 3D geometry. The shadow is 
simply a "flattened-out" replica of all the lines (without the vertical component and 
translated to the ground plane) drawn with a thicker linewidth, and in a shade slightly 
darker than the ground color. Since no Z-buffering is used, the shadow is drawn first, 
followed by the sticks. 

For the creatures with variable thickness in parts, a number of techniques have 
been used, many of them are graphics hacks meant to reduce polygon-count or to 
avoid expensive Z-buffering and lighting algorithms. Since these are specifically 
alife-oriented worlds, without any objects besides abstract articulated bodies, specific 
graphics techniques are used. At the point in which these creatures are imbedded in a 
more comprehensive virtual world, a more generalized renderer would be useful. 

Rendering Behavior, Then Pixels. Part of the methodology is to take advantage of 
something that the eye-brain system is very good at: detecting movement generated 
by a living thing. It is important not to spend too much computational time on 
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heavyweight texturemaps, lighting models, and cute cosmetics, when more computa- 
tion can be spent on deep physics and animation speed (to show off the effects of 
subtle movements that have evolved). It is a purist philosophy: simply show what is 
there. No pasting 2D images of eyes and ears on the creatures. Instead, as the simula- 
tion itself deepens (for instance, when light sensors or vibration sensors are allowed 
to evolve on arbitrary parts of bodies, as evolution dictates) graphical elements visu- 
alizing these phenotypic features will be rendered. They may be recognizably eye and 
ear-like, or they may not. 



8 Commentary: Why Are There No Humanoids Here? 

The goal of Artificial Intelligence research is something magnificent, and something 
very human indeed. It appears as if the skills enabling one to play chess and use 
grammar are a very recent evolutionary trend, whereas climbing a tree is a much 
older skill, truly more complex, and in fact possibly an activity whose basic cognitive 
mechanisms are the primordial stuff of higher reasoning. 

These simulations do not aim to model humans. But if that were the goal, it may be 
best to design a vestibular system and an environment in which vertical, bipedal lo- 
comotion might be advantageous. Best to design an evolvable neural system (and to 
allocate lots of computer memory!) and a highly complex, dynamic environment, 
whereby something brain-like might billow like a mushroom cloud as an internal echo 
of the external complexity. Best to design a simulation in which objects existing in 
the world might be exploited as tools and symbols, extended phenotypes, cultural 
memes. What might evolve then could be something more human-like than the wig- 
gling worms shown in the illustrations of this paper. But in the meanwhile, there is no 
rush on my part to get to human levels with these simulations. And at any rate, imagi- 
nary animals can be very thought-provoking. 



9 Conclusion 

In this paper, I have described many of the techniques, concepts, and concerns in- 
volved in a methodology for creating animated artificial life worlds. This approach 
does not begin with a biological research agenda but rather an eye towards the crea- 
tion of forms and motions which are lifelike and autonomous-a branch of character 
animation. The alife agenda of studying emergent behavior by way of the crafting of 
simulations has become incorporated into this animation methodology, and with it, 
some key themes from evolutionary biology, such as sexual selection. It is hoped that 
these concepts and techniques will be useful for further advances in animated artifi- 
cial life, for education, entertainment, and research. 
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Abstract Artificial life might come to play important roles for the 
World Wide Web, both as a source of new algorithmic paradigms and 
as a source of inspiration for its future development. New Web searching 
and managing techniques, based on artificial life principles, have been 
elicited by the striking similarities between the Web and natural envi- 
ronments. New Web interface designs, based on artificial life techniques, 
have resulted in increased aesthetic appeal, smart animations, and clever 
interactive rules. In this paper we exemplify these observations by sur- 
veying a number of meeting points between artificial life and the Web. 
We touch on a few implementation issues and attempt to draw some 
lessons to be learned from these early experiences. 



1 Introduction 
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rom t nvironm nt w 11 int rn 1 t t p r orm om omput tion n 
X ut tion . tion v nnr otutm r ult in n r int . n- 

r i u up n umul t int rn 11 t rou out n nt li ; it int rn 1 

1 V 1 utom ti 11 t rmin r pro u tion n t v nt in w i nr 

i on rv . nt t t p r orm tt ttrtnvr r pro u mor 

n oloniz t popul tion. n ir t int r tion mon nt o ur wit out 

t n o xp n iv ommuni tion vi omp tition or t r nit n- 

vironm nt 1 r our . Mut tion n ro ov r or t n n r or 
t volution o n mi 11 pt nt . i p r i m n or n it - 

p n nt 1 tion t xp t popul tion iz i t rmin t rr in 
p it o t nvironm nt. o i tin i nr o t wit xp n iv 
tion intrin i 11 n or In n twor lo limitin in i nt u 

o n wi t . 

Collective Behavior pt tion m n or nt to on ntr t in i n- 
r rot wrmno um nt r r 1 v nt. nt urviv 1 

will n ur x n in n qu t flow o in orm tion or n r . 

itu tion i illu tr t t n p ot in i ur 1 illu tr tin t pi 1 oil - 

tiv vior o n o pi r in r pon to qu r n limit to w 11 n 

un o t . t r w il lu nt v oun w t t u r w nt ; 
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t pro p r n multipl in t i r 1 v nt ni w il ot r nt ontinu to 

xplor t worl in r o It rn tiv . 




Figurel. 
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Search Efficiency On o t ntr 1 r ult o t n o pi r proj t i to 

V own t t it i u ul or in orm tion r 1 orit m to vi w t 

n tur 1 nvironm nt n r t riz it in u t rm . t r t n- 

vironm nt 1 t rom 1 rnin nt p r p tiv ? t ti ti 1 tur u 
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t t lin topolo n in t t n xploit i tri ut nt 

outp r ormin x u tiv r n or r o m nitu . Mor ov r t n- 

r tw n volution 1 rnin nrlvn inu n ition 1 

our- ol 00 t in p r orm n 15 . 

3 Applications to Web Interface Design 

nt r i n ( ) i r ti 11 n in in it p t n un tion- 

lit . t it i un 1 r in w t ir tion it i movin . r r t 1 t t r 

qu lit tor in ( ) 1 v r int r tiv rul ; ( ) m rt nim tion; n ( ) 

t ti pp 1.^ ntil r ntl int r tivit n m rt nim tion n - 

m or i rti i 1 int Hi n t niqu r li onl on nim tor or 

pro r mm r iliti ; n t t ti pp 1 o r li uniqu 1 on 

r p i i n r iliti . ow v r in u r w nt to t out o t i un x it- 
inln ptri mnor mor pp lin or rti ti t ti mu 
m rt r — i not li -li — nim tion n mor 1 v r — i not um n-li — 

int r tion . Li t niqu pp r mon t rli t n mo t pi u i 1 n- 

i t to r om ot n .n int rminroti tion 
w ow om X mpl in w i Li Ip u to ul 11 1 n provi in 

or n w w to int r t ( ) m in nim tion ri r n mor n tur 1 
( ) n xt n in t um n p iliti to uil t ti rti t ( ). 

3.1 Web Interactive Cellular Automata 

t n or nt r tiv llul r utom t . t i m o Li 

13 (or n ot r) llul r utom ton 20 t t i ontroll r flowin t xt 

on n ML p . r flowin t xt i om tiv t xt t t w n li 

nmill n t ttot llul r utom t . t w initi 11 r t 

to upport - inn provi pi in n in n unpr i t 1 ni- 
m tion or p lo o . t w t n xt n to upport n mi omput r 
rt n ot r in o ppli tion . om x mpl o n n on t 

om p 3 . nim tion m to Imo t r quir m nt in t ur- 
r nt ion o p ; om t in mu t Iw mov . t p i t ti 
it om orin . r or w lot o nim tion on t n t. Mo t o 

^ Here we will not take into account other important, but secondary factors, such as 
user-oriented interface personalization, interface legibility, multilevel functionality, 
etc. 
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Figure2. 



n mi w it lo o. 



p i impl m nt tion w m o t t t llul r utom - 

ton ppl t u onw 1 i m o Li rul . ow v r t worl 

(impl m nt r nr itr r roun pi tur ) i r two p r- 

11 1 in p n nt ov rl ppin . t i n int r tiv ppl t w r ontrol 

i i V t rou li in on t xtu 1 lin r t r t n t ort o ox u 
o utton n m nu . tu 1 im n ion o t ri r r itr r ut 

in u u 11 o wit pi tur roun t iz i i t t t 

pi tur . 

1 orit m impl m nt t r m t o or m in it 1 ul tion 

i ntl ^ 

1. il t urr nt nr tion i i pi on t r n t n xt nr tion 
i pr p r on n o - r n u r n t n r pi t urr nt r n in 

in 1 t p. 

2. Onl 11 t t n t ir t tu r r -p int 



^ The basic algorithm is due to Andreas Ehrencrona [5]. 
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3. n 1 ul tin t num r o liv n i or o 11 nowl rom 

pr viou 11 1 ul tion in . . in t It olumn o t urr nt 

Hit mi 1 olumn o t pr viou 11 t t tu o t 1 t n i or 
i nown n n not r 1 ul t 

n mi r t r o i on two p r 11 1 in p n nt 

pi in imult n on 1 in t m worl . worl i ontroll v - 

ript t t m nt t t tiv t v routin . n t i routin i tiv t 
n w p tt rn o 11 i in rt into on o t . t t r iv t 

p tt rn i It t r n om. 1 o It t r n om r tr prmtr 

o t n w p tt rn t p tt rn it 1 it olor n it lo tion wit in t 

r m . p tt rn it 1 i It rom li t o t r int r tin n w 11- nown 
un t 1 p tt rn t R-Pentomino t H-Heptomino n t Pi-Heptomino. 
olor i It t r n om rom li t n in t ML 1 . 

i impl m nt tion r ult in vin x tl two olor ( n two popul - 

tion ) t n iv n mom nt ut t olor m n ivin t illu ion o 

in n w popul tion . t 1 o iv om w t un xp t vior on 

o t popul tion u nl n olor n t n w p tt rn. 

Text Oriented Interactivity nim tion n int r t wit xt 

Ori nt nt r ( O ) 17 o t t t Iw r n w t m Iv . oi 6 
own in i ur 3 i u n xt n ion o n not r x mpl o v lu - 

ommuni tion. oi provi t u r wit n int r tiv m o Li 

impl m nt tion t t i ontroll r -flowin t xt. w i n to 

r t in p r 11 1 to r ul r ML p rt xt v nt in ort o -pro u t to 
r ul r p r qu t n Iw pro u in t m ( 1 it unpr it 1 ) 
vior n m 1 in n w p tt rn to t worl . oi on t 

ot r n i in to p r orm v riou pr n tion . n oi v riou 
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oi m u o t r tool ML r m v n v ript. ML 
r m r u or r tin p w r on p rt o t p (t pro r m or t 
pi tur ) i t ti n on t ntl i pi in pr n lo tion w il t r t 
o t p n roll or n ont nt. i m t iz o t u r 
int r pr ti 11 unlimit . v i u or t pro r m it 1 v ript 
or m nipul tin t p rt xt lin n t v - v ript int r i u 
to How t t xt to ontrol t pro r m. 

t xt ri t ppl t t t i i pi on t ML p n 

Ilow t row in u r to pi n 1 rn wit it li in on t ppropri t 
wor . oi ppli tion t u mon tr t t t Li t niqu n 

u tion 1 V lu to t m in r in n int r tiv n liv 

pro . or X mpl ou n im in n rti i 1 Li nt r tiv n oo 

w r u in t i t niqu ou n tu 11 tr n orm our r in into n 

tiv pro . n x mpl woul i ou o not now w t in unit in 
n ur 1 n twor i ou ju t li on t i n unit t xt n ou t t t 
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Figure4. 
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® An approximate neighbor detection algorithm is used for efficiency, which is of course 
a crucial aspect in Web design. 
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w i pi in i t in orm tion r n. r non- volvin lo 

liv t rn 11 (or until t ir n r rop to z ro) lo liv on nr tion 
n nr tion n w n t tr n r i n 11 ill . lo volv xu 11 
w r lo i t n nt o two p r nt . two p r nt r It 

or in to two tn ttri nt nr n t . n lo n in or lo 
t urin it li tim n t mor it t tt r it i . n r i lo t 

proportion 11 to p n t i in rom proximit to n i or . it 
t i t-up on n il m xp rim nt wit popul tion o lo vin 
pr n vior or p r orm imul tion o t volution r pro tunin 

t r 1 tiv w i t o t n nr in t tn un tion. 



3.3 Interactive Computer Art 

lo V X mpli ow Li mo 1 n impl m nt to run ro t 
ut o u t m or n t in u ul rom point o vi w? 

po itiv n w r m o t in on i rin int r tiv omput r rt w r 

t u r int r t wit rpilojtrt rt no . rti il int r 

pro r m 1 21 10 i on o v r 1 ppli tion o Li to omput r r p i 
i n. t i on rti i 1 n ur 1 n twor n volution r 1 orit m . t 

t innin o volution r ion it n r t 16 r n om im w i 

r ort o -im or t p int r to t rt it wor . u r n 1 t 

our otm prntwi tn pro u ixt n n w in ivi u 1 on titutin 
n w nr tion. u r n zoom out o t im r nil loo 

ttmnorv n trn n int r tion in ompo ition olor 
t xtur orm n p r p tiv . onl prmtrt ur n ontrol t 
n r tion i mut tion r t (w i ow v r n in i r ntl or 

n r tion). 




Figures. 



li liv rt w it . 
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ppli tion o t i pro rm n nin rnui om p 11 

n t li liv rt it 12 own in i nr 5. it mon tr t 

oupl o point . ir t Li t niqu n ppli to r p i inn 
t r or to in t o rnui it . on t u r n 

int r t wit n influ n pi tur ontrollin it Li - volution. 

li liv rt it riv rom t rti i 1 int r n lo How 
t u r to o o int r tin wit t t xt t t i tt to n p rt o 
t pi tur . 



4 Conclusion 
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Information Flocking: Data Visualisation in Virtual 
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Abstract. A novel method of visualising data based upon the schooling 
behaviour of fish is described. The technique allows the user to see complex 
correlations between data items through the amount of time each fish spends 
near others. It is an example of a biologically inspired approach to data 
visualisation in virtual worlds, as well as being one of the first uses of VRML 
2.0 and Java to create Artificial Life. We describe an initial application of the 
system, the visualisation of the interests of a group of users. We conclude that 
Information Flocking is a particularly powerful technique because it presents 
data in a colourful, dynamic form that allows people to easily identify patterns 
that would not otherwise be obvious. 



1. Introduction 



1.1 Flocking & Schooling 

Birds flock, fish school, cattle herd. The natural world has many examples of species 
that organise themselves into groups for some reason, for example to reduce 
predation. It has been shown [1] that many predators are "tuned" to hunting 
individuals, and are confused by large numbers of animals organised into a flock or 
school. Although the evolutionary advantages in flocks had been well characterised 
no simple models reproducing such behaviours had been demonstrated. 

Reynolds created computer simulations of such flocks by modelling a few simple 
rules [2], and christened individuals that undergo such flocking "boids". Animations 
based upon boid-like motion have appeared in a number of Hollywood films'. 

The emergent behaviour of the flock is the result of the interaction of a few simple 
rules. In Reynolds' simulation, these were: 



' Examples: herds of dinosaurs in “Jurassic Park”, flocks of bats in “Cliffhanger” and “Batman 
Returns” and the wildebeest stampede in “The Lion King”. 

J.-C. Heudin (Ed.): Virtual Worlds 98, LNAI 1434, pp. 168-176, 1998 
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• collision avoidance 

• velocity matching (flying with the same speed and direction as the others) 

• flock centring (trying to fly near the centroid of one's neighbours) 

These rules are sufficient to reproduce natural behaviours, particularly if a predator 
is treated as an obstacle. However their simplicity allows the use of such self- 
organised behaviour to be extended to serve a more useful purpose: data visualisation. 
In Information Flocking, a fourth rule is added which modifies the motion of the 
individuals on the basis of some similarity measure. This measure can be derived 
from a set of data, with each individual bold associated with a single data item. The 
flocking motion then becomes a means of visualising the similarities between the 
individual data items. 

A virtual world was created to display the flocking behaviour. Initially this consists 
of a school of fish swimming around in 3D, but it is easily extended to include such 
concepts as attractive and repellent objects (which might attract or repel specific 
items), and predators which might act as data filters. 

The initial problem to which Information Flocking was applied is that of 
visualising the interests of a group of people. Previously, hierarchical clustering 
techniques [5] have been applied to such data sets. Neural network approaches have 
also been used [6]. In particular, Orwig and Chen [7] have used Kohonen neural nets 
[8] to produced graphical representations of such data. While these representations 
proved to be much faster and at least as powerful as subjective human classification, 
they are essentially static. Information flocking is dynamic in that the fish in the 
simulation can change their behaviour in response to changes in the underlying data 
set. The output is also dynamic (the fish “swim”) which allows the human viewer to 
identify patterns more easily. 



2. Methods 



2.1 VRML 

The prototype Information Flocking system was developed using VRML (Virtual 
Reality Modelling Language), version 2.0^. This is a powerful, emerging technology 
that allows rapid development of graphical programs. Objects in a VRML "world" 
can be controlled by means of a Java script. This results in a system that can produce 
3D, interactive graphical simulations that can be controlled using all the features of 
the Java programming language. VRML is also platform independent and allows easy 
creation of multi-user environments. A schematic of the system is given in Fig. 1. 



^ See the VRML Repository at http://www.sdsc.edu/vrml/ for more information on VRML. 
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Fig. 1. Schematic of a VRML world. The world is organised hierarchically; different node 
types define geometry, appearance, camera views etc.. A Script node allows interfacing to a 
Java program. This controls the behaviour of the fish. Communication is event-driven, and 
events are connected hy structures known as Routes. Objects (e.g. fish) can be “anchored” to 
URLs; thus when a fish is clicked upon, the relevant page can be opened in a normal web 
browser such as Netscape. A more detailed description of VRML can be found at the VRML 
Repository (see footnote on previous page). 



2.2 Modelling Emergent Behaviour 

The fish in the demonstration are initially positioned randomly. They then swim 
along a heading vector, which is the resultant of several component vectors, one for 
each of the behavioural rules. In the current system, these are: 

• collision avoidance - avoid collisions with other fish if possible. The vector for this 
behaviour is calculated to take each individual away from its close neighbours. 

• velocity matching - each individual tries to swim in the same direction and at the 
same speed as nearby fish, subject to a minimum and maximum speed. 
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• origin homing - try to swim towards the origin. This has the effect of keeping the 
fish on screen, and results in their swimming around in a simulated "bowl", rather 
than disappearing into infinity. 

• information flocking - swim closer to fish that are similar, and further from fish 
that are different^ The vector for this is calculated as the weighted resultant of all 
the vectors between the current fish and its neighbours. The weights are obtained 
from the correlation matrix described in §2.3. 

The result of these calculations is a vector representing the ideal heading for the fish 
in question. This ideal heading vector is averaged with the current heading to give the 
fish’s actual new heading. 

The calculation of the heading vector for a fish A is as follows: 

New heading = - A) + w^„Z(C - A) + - O) + w,pZs/^’,. - A) 

where: 

w^A> Wqjj, Wjp = weighting applied to Collision Avoidance, Velocity Matching, 
Origin Homing and Information Flocking behaviours respectively 
A, b., O = position vector of fish A, fish i and origin respectively 
C = position vector of the centroid of fish A’s neighbours (see below) 

Sjj = similarity of fish A and fish i. 

Notes: 

1 . The collision avoidance component of the heading vector is repulsive, while all the 
other components are attractive. 

2. The similarity matrix is calculated at the start of the simulation, to maximise the 
speed. There is no reason that it could not be calculated on-the-fly from “live” 
data; this would result in the fish changing their behaviour dynamically to reflect 
the changes in the data. 

3. At each time-step, two main calculations must be performed: 

i. The matrix of distances between the fish must be recalculated. This matrix is 
symmetric, so in order to optimise the speed of the simulation, only half of it is 



^ The weight, Wjp, applied to the Information Flocking behavioural component, determines the 
strength of inter-fish attraction. In the “interest” data set that is used for the current 
demonstration, w,p lies between 0.0 and 1.0, so completely dissimilar fish have zero 
attraction. It would be feasible for Wjp to be in the range -1.0 to 1.0; this would result in a 
more marked separation. The “interest” data set was used as obtained by the authors, without 
modification. 
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calculated and then copied into the other half (since the distance between fish i 
and fish j is equal to the distance between fish j and fish /!) Distances are 
actually stored as their squares in order to avoid large numbers of calculations 
of computationally-intensive square roots. 

ii. The near-neighbours of each fish must be recalculated. This is based on the 
distance matrix described above. 

Each fish can sense its environment to a very limited extent. They can "see" for a 
fixed distance in a sphere around them. No attempt has been made to implement 
directional vision; the simple spherical model was considered sufficient for our 
purposes. The distance that the fish can "see" can be varied; in most of our 
investigations so far a distance approximately equal to four body lengths has been 
appropriate. This point is discussed in more detail below. 

Each fish can be given a colour and a label, which appears when the mouse pointer 
is moved over it. URLs (Uniform Resource Locators) can be specified for each fish; 
when this is done, clicking on the fish will open the relevant web page in a browser 
such as Netscape. 

It should be pointed out that the choice of the fish representation was an arbitrary 
decision based upon ease of construction and rendering. The behaviour of birds, bees 
or sheep could be modelled with very little modification. Other workers, notably 
Terzopoulos, Tu and Grzeszczuk have presented work that concentrates on the 
simulation of the behaviour of real fish [3]. 



2.3 Data Used 

The data used for the Information Flocking needs to provide weights for all possible 
pairwise interactions between the fish. Thus, it needs to be in the form of a square, 
symmetric matrix. In the current application, this is in the form of a matrix of 
similarities between the interests of a group of Internet users [4], but any data in the 
correct format could be visualised using this technique. 

The colours, labels and URLs of the fish are also read in from files. In the case of 
the ‘interest’ data, the labels represent the interests of the individuals, and the colour 
of each fish represents the primary interest of that person. 




Information Flocking: Data Visualisation in Virtual Worlds 



173 



3. Results 

An example screenshot is shown in Fig. 2. 




Fig. 2. Screenshot of the Information Flocking simulation. The labels represent the colours of 
the fish when seen on a colour monitor. Only the colours referred to in the text below are 
labelled. The ellipses in the figure serve only to delineate the colour groups; they do not appear 
in the actual simulation. 

Watching the simulation for even a short period of time is very instructive. Several 
closely-related groups are immediately visible; the green, white and dark blue fish 
form small groups, indicating that the users in each of these groups have very similar 
interests. However, the individual groups also tend to form larger, more diffuse 
“supergroups”; for example the green, white, dark blue and purple fish all tend to 
move around in a large group, indicating a lower, but still significant, level of inter- 
group similarity. 

Other small groups remain distinct from the main group. The light blue fish tend to 
exhibit markedly different behaviour, indicating that the users whom they represent 
have substantially different interests to the other users. 
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More subtle interactions can also be seen. The two yellow fish ostensibly represent 
people with the same primary interest, yet they do not swim close to each other. Upon 
further inspection, the two individuals turn out to be interested in different aspects of 
the same subject, which is the reason for the separation of the fish representing them. 



4. Future Developments 

Feedback on the prototype information flocking demonstrator described in this paper 
has been exceedingly positive; for this reason we are actively developing the system. 
Development is proceeding in three main areas: 

1. Speed: currently the system runs at an acceptable frame rate (30 frames per second 
on a high-end PC with hardware 3D acceleration) for data sets represented by up to 
about 250 fish. Clearly this needs to be extended to handle data sets involving 
1000 or more individuals (see note 2 below). Recent advances in VRML browsers, 
rendering engines and hardware graphics acceleration are also yielding 
considerable speed increases. 

2. Data sets: we are investigating the properties of the system when applied to data 
sets other than the one initially used. Other correlation methods are also being 
tried. 

3. Statistical investigations: through initial experiments with the current Information 
Flocking demonstrator, it appears that several factors can significantly affect the 
emergent behaviour. For example: 

• the weights applied to each of the component vectors in the calculation of the 
heading vector. 

• the distance within which the fish can detect each other 

Preliminary results from some of these further developments are very illuminating. 
For instance, when a simulation with over 300 fish was set-up it was found that if the 
vision horizon was too long they formed a single, undifferentiated clump. If it was 
too short they never flocked but moved as individuals. There is clearly an optimal 
setting which leads to the formation of fluid groups, that keep partially breaking up 
and reforming. The suggestion here is that this phenomenon may be explained as a 
phase transition between a gaseous random state and a solid one, as the order 
parameter is changed. Clearly the system needs to be tuned into the phase transition 
zone to work best. Such phenomena are also seen in classical hierarchical clustering 
[5], and work is underway to reconcile these explanations. 

As the exact values depend on all the parameter settings, and on the numbers of 
fish, their relative volume and that of the virtual space this could be difficult. We are 




Information Flocking: Data Visualisation in Virtual Worlds 



175 



now developing a system where the fish increase or decrease their horizon depending 
on their ‘satisfaction’ - the degree to which they have found similar fish near them. 
Early results suggests this leads to some particularly interesting phenomena akin to 
courtship behaviours. 



5. Conclusions 

This paper has described the concept of Information Flocking. It shows that ideas in 
artificial life, developed to mimic biology have much more powerful applications 
outside the arena of simulations. 

The technique is powerful because it presents data in a form which is particularly 
suited to the human brain’s evolved ability to process information. Human beings are 
very good at seeing patterns in colour and motion, and Information Flocking is a way 
of leveraging this ability. The dynamics of the system provide a much greater degree 
of understanding that the initial data would suggest. 

Virtual worlds with virtual organisms could have a powerful impact on the way we 
use and visualise complex data systems. A number of demonstrators have now been 
built that show these concepts in action. 
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Abstract. Nerve Garden is a biologically-inspired multi-user collaborative 3D 
virtual world available to a wide Internet audience. The project combines a 
number of methods and technologies, including L-systems, Java, cellular 
automata, and VRML. Nerve Garden is a work in progress designed to provide 
a compelling experience of a virtual terrarium which exhibits properties of 
growth, decay and cybernetics reminiscent of a simple ecosystem. The goals of 
the Nerve Garden project are to create an on-line “collaborative Artificial Life 
laboratory” which can be extended by a large number of users for purposes of 
education and research. 



Introduction 

During the summer of 1994, one of us (Darner) paid a visit to the Santa Fe Institute 
(SFI) for discussions with Chris Langton and his student team working on the Swarm 
simulation system [8]. Two fortuitous events coincided with that visit: SFI was 
installing the first World Wide Web browsers, and digital movies of Karl Sims’ 
evolving “block creatures” [1, 7] were being viewed through the Web by amazed 
students (figure 1). It was postulated then that the combination of the emerging 
backbone of the Internet, a distributed simulation environment like Swarm and the 
compelling 3D visuals and underlying techniques of Sims’ creatures could be 
combined to produce something very compelling: on-line virtual worlds in which 
thousands of users could collaboratively experiment with biological paradigms. 

In the three years since the SFI visit, we founded an organization called the Contact 
Consortium [11]. This organization serves as a focal point for the development of on- 
line virtual worlds and hosts conferences and research and development projects. One 
of its special interest groups, called Biota.org, was chartered to develop virtual worlds 
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using techniques from the natural world. Its first effort is Nerve Garden which came 
on-line in August of 1997 at the SIGGRAPH 97 conference [12]. 




Fig. 1. View of two of Karl Sims’ original evolving creatures in a competition to control a 
block. Iterations of this exercise with mutations of creature’s underlying genotypes yielding 
winning strategies created a large variety of forms. Above we see a two armed grasping 
strategy being bested by a single armed ‘hockey stick’ strategy. Courtesy K Sims. See [7] for 
the original MPEG movies of Sims’ creatures. 

Three hundred visitors to the Nerve Garden installation used L-systems and Java to 
germinate plants models into a shared VRML (Virtual Reality Modeling Language) 
world hosted on the Internet. Biota.org is now developing a subsequent version of 
Nerve Garden, which will embody more biological paradigms, and, we hope, create 
an environment capable of supporting education, research, and cross-pollination 
between traditional Artificial Life subject areas and other fields. 



Nerve Garden I: Architecture and Experience 



Nerve Garden I is a biologically-inspired shared state 3D virtual world available to a 
wide audience through standard Internet protocols running on all major hardware 
platforms. Nerve Garden was inspired by the original work on Artificial Life by Chris 
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Langton [3], Tierra and other digital ecosystems by Tom Ray [2,3] and the evolving 
3D virtual creatures of Karl Sims [1]. Nerve Garden sources its models and methods 
from the original work on L-systems by Aristide Lindenmayer, Przemyslaw 
Prusinkiewicz and Radomir Mech [5, 6] and the tools from Laurens Lapre [10]. 




Fig. 2. Lace Germinator Java client interface showing the mutator outputs on the left, 
the growth and naming interfaces at the bottom with a plant production visualized in 
the main window. 

The first version. Nerve Garden I, was built as a client-server system written in the 
distributed Java language. The client program, called the Germinator (figure 2), 
allowed users to extrude 3D plant models generated from L-systems. The simple 
interface in the Java client provided an immediate 3D experience of various L-system 
plant and arthropod forms. Users employed a slider bar to extrude the models in real 
time and a mutator to randomize production rules in the L-systems and generate 
variants on the plant models. Figure 3 shows various plant extrusions produced by the 
Germinator, including models with fluorescences. After germinating several plants. 
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the user would select one, name it and submit it into to a common VRML 2.0 
scenegraph called the Seeder Garden. 




Fig. 3. Plant models generated by the Germinator including flowering and vine like 
forms. Arthropods and other segmented or symmetrical forms could also be 
generated. 

The object passed to the Seeder Garden contained the VRML export from the 
Germinator, the plant name and other data. The server-based Java application, called 
NerveServer, received this object and determined a free “plot” on an island model in 
a VRML scenegraph. Each island had a set number of plots and showed the user the 
plot his or her plant could be placed by moving an indicator sphere operated through 
the VRML external authoring interface (EAI). “Cybergardeners” would open a 
window into the Seeder Garden where they would then move the indicator sphere 
with their plant attached and place it into the scene. Please see an overview of the 
client-server architecture of Nerve Garden I in figure 4. 

Various scenegraph viewpoints were made available to users. Including a moving 
viewpoint on the back of an animated model of a flying insect endlessly touring the 
island. Lfsers would often spot their plant as their bee or butterfly made a close 
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approach over the island (figure 5). Over 10MB of sound, some of it also generated 
algorithmically, emanated from different objects on the island added to the immersion 
of the experience. For added effect, L-system based VRML lightening (with 
generated thunder) occasionally streaked across the sky above the Seeder Garden 
islands. The populated seeder island was projected on a large screen at the Siggraph 
hall which drew a large number of visitors to the installation. 
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Fig. 4. Nerve Garden I architecture detailing the Java and VRML 2.0 based client- 
server components and their interaction. 

NerveServer permitted multiple users to update and view the same island. In addition, 
users could navigate the same space using standard VRML plug-ins to Web browsers 
on Unix workstations from Silicon Graphics, PCs or Macintosh computers operating 
at various locations on the Internet. One problem was that the distributed L-system 
clients could easily generate scenes with several hundred thousand polygons, 
rendering them impossible to visit. We used 3D hardware acceleration, including an 
SGI Onyx II Infinite Reality system and a PC running a 3D Labs Permedia video 
acceleration card to permit a more complex environment to be experienced by users. 




Fig. 5. Flight of the bumblebee above a Seeder Garden 
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In 1999 and beyond, a whole new generation of 3D chip sets on 32 and 64 bit 
platforms will enable highly complex 3D interactive environments. There is an 
interesting parallel here to Ray’s work on Tierra [2,9], where the energy of the system 
was proportional to the power of the CPU serving the virtual machine inhabited by 
Tierran organisms. In many Artificial Life systems, it is not important to have a 
compelling 3D interface. The benefits to providing one for Nerve Garden are that it 
encouraged participation and experimentation from a wide non-technical group of 
users. The experience of Nerve Garden I is fully documented on the Web and several 
gardens generated during the SIGGRAPH 97 installation can be visited on-line with a 
VRML 2.0 equipped web browser [12]. 



What was Learned 

As a complex set of parts including a Java client, simple object distribution system, a 
multi-user server, a rudimentary database and a shared, persistent VRML scenegraph, 
Nerve Garden functioned well under the pressures of a diverse range of users on 
multiple hardware platforms. Users were able to use the Germinator applet without 
our assistance to generate fairly complex, unique, and aesthetically pleasing models. 
Users were all familiar with the metaphor of gardens and many were eager to “visit 
their plant” again from their home computers. Placing their plants in the VRML 
Seeder Gardens was more challenging due to the difficulty of navigating in 3D using 
VRML browsers. Younger users tended to be much more adept at using the 3D 
environment. 

In summary, while a successful user experience of a generative environment. Nerve 
Garden I lacked the sophistication of a “true -Life system” like Tierra [2,9] in that 
plant model objects did not reproduce or communicate between virtual machines 
containing other gardens. In addition, unlike an adaptive L-system space such as the 
one described by Mech and Prusinkiewicz [6] the plant models did not interact with 
their neighbors or the environment. Lastly, there was no concept of autonomous, self 
replicating objects within the environment. Nerve Garden II, now under development, 
will address some of these shortcomings, and, we hope, contribute a powerful tool for 
education and research in the ALife community. 



The Next Steps: Nerve Garden II 

The goals for Nerve Garden II are: 

• to develop a simple functioning ecosystem within the VRML scenegraph to control 
polygon growth and evolve elements of the world through time; 

• to integrate with a stronger database to permit garden cloning and inter-garden 
communication encouraging cross pollination between islands; 
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• to integrate a cellular automata engine which will support autonomous growth and 
replication of plant models and introduce a class of virtual herbivores 
(“polyvores”) which prey on the plants’ polygonal energy stores; 

• to stream world geometry through the transmission of generative algorithms (such 
as the L-systems) rather than geometry, achieving great compression, efficient use 
of bandwidth and control of polygon explosion and scene evolution on the client 
side; 

Much of the above depends on the availability of a comprehensive scenegraph and 
behavior control mechanism. In development over the past two years. Nerves is a 
simple but high performance general purpose cellular automata engine written as both 
a C++ and Java kernel [13]. Nerves is modeled on the biological processes seen in 
animal nervous systems, and plant and animal circulatory systems, which all could be 
reduced to token passing and storage mechanisms. Nerves and its associated 
language, NerveScript, allows users to define a large number of arbitrary pathways 
and collection pools supporting flows of arbitrary tokens, token storage, token 
correlation, and filtering. Nerves borrows many concepts from neural networks and 
directed graphs used in concert with genetic and generative algorithms as reported by 
Ray [3], Sims [1] and others. 

Nerves components will underlie the Seeder Gardens providing functions analogous 
to a drip irrigation system, defining a finite and therefore regulatory resource from 
which the plant models must draw for continued growth. In addition. Nerves control 
paths will be generated as L-system models extrude, providing wiring connected to 
the geometry and proximity sensors in the model. This will permit interaction with 
the plant models. When pruning of plant geometry occurs or growth stimulus 
becomes scarce, the transformation of the plant models can be triggered. One step 
beyond this will be the introduction of autonomous entities into the gardens, which 
we term “polyvores”, that will seek to convert the “energy” represented by the 
polygons in the plant models, into reproductive capacity. Polyvores will provide 
another source of regulation in this simple ecosystem. Gardens will maintain their 
interactive capacity, allowing users to enter, germinate plants, introduce polyvores, 
and prune plants or cull polyvores. Gardens will also run as automated systems, 
maintaining polygon complexity within boundaries that allow users to enter the 
environment. 

Example of a NerveScript coding language 

spinalTap . nrv : 

DEF spinalCordSeg Bundle { 

-spinalTapA-Swim-bodyMotion [ 4 ] -Complex; 
-spinalTapB-Swim-bodyMotion [ 4 ] -Complex; 

} 
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We expect to use Nerves to tie much of the above processes together. Like VRML, 
Nerves is described by a set of public domain APIs and a published language, 
NerveScript [13]. Figure 6 lists some typical NerveScript statements which describe a 
two chain neural pathway that might be used as a spinal chord of a simple swimming 
fish. DBF defines a reusable object spinalCordSeg consisting of input paths 
spinalTapA and spinalTapB which will only pass the token Swim into a four 
stage filter called bodyMotion. All generated tokens end up in Complex, another 
Nerve bundle, defined elsewhere. 




Fig. 6. Nerves visualizer running within the NerveScript development environment. In the 
VRML setting, the pathways spinalTapA and spinalTapB are fed by eventOut messages drawn 
out of the scenegraph while the Nerve bundles generate eventins back to VRML using the EAI. 
BodyMotion are a series of stages where filters draw off message tokens to drive simulated 
musculature on the left and right hand side of the VRML model, simulating a swimming 
motion. Message tokens that reach Complex, a kind of simple brain, trigger events in the 
overall simulation, and exporting messages back into the external pool. 

Figure 6 shows the visualization of the sample NerveScript code running in the 
NerveScript development environment. We are currently connecting Nerves to the 
VRML environment, where it will be possible to visualize the operation in 3D. 
Nerves is fully described at the web address referenced at the end of this paper. 



Goals and Call for Participation 

The goals of the Nerve Garden project are to create an on-line “collaborative A-Life 
laboratory” which can be extended by a large number of users for purposes of 
education and research. We plan to launch Nerve Garden II on the Internet and look 
forward to observing both the user experience and, we hope, the emergence of 
complex forms and interactions within some of the simple garden ecosystems. The 
Contact Consortium and its Biota.org special interest group would welcome 
additional collaborators on this project. 
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Abstract. In this study, we introduce a remarkable squared tiling of 
the plan whose characteristics meet in many points those of the genetic 
code: same number of structural levels (3), same number of elements at 
each level (4, 64 and 20), same relationships between the elements of the 
different levels. To conclude, we formulate one hypothesis to explain these 
results and consequently we propose new ways of research in Artificial 
Life, but also in structural molecular biology. 



1 Introduction and Purpose 

In Artificial Life studies. Mathematics and mathematical tools are important to 
explore and to experiment (by means of computer simulations) the behaviour 
and the fundamental structures of artificial living systems. 

In real living systems studies, biological experiments are essential. In this 
domain, some recent experiments led by James Ferris (in collaboration with 
Leslie Orgel of the Salk Institute) show the importance of mineral surfaces^ and 
- more precisely - the importance of geometrical constraints in the early stages 
of bio-molecular development (i.e. at the origin of life): mineral surfaces restrict 
the elementary molecular movements into a weak number of particular directions 
and consequently favorize the molecular polymerization (and polycondensation) 
processes on these surfaces [1]. 

Purpose: we propose to study, in a virtual mathematical plan, i.e. in a two 
dimensional virtual world, the consequences of geometrical constraints that have 
been observed in the real world (physico-chemical restriction of the directions of 
molecular movements). 



^ The hypothesis of the origin of life on mineral surfaces had remained, longtime, with- 
out determining clue. Now, the experimental works of James Ferris and al. bring some 
fundamental and decisive elements to satisfy to this hypothesis: montmorillonite sur- 
face for nucleotides polymerization (ImpA) and illite and hydroxy lapatite surfaces for 
amino acids polymerization (glutamic and aspartic acids). 
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2 Simplest Tiling of the Ideal Mathematical Plan 

2.1 Tiling by Squares Requires the Weakest Number of Tiled 
Directions 

In the ideal mathematical plan, which is the equivalent representation of an 
elementary bio-molecule? 

In response, we can choose an elementary piece of the plan surface: for exam- 
ple, a polygonal tile. To link these artificial bio-molecules (the linking of tiles is 
similar to a polymerization process), it is necessary to be able to pave the plan 
with the same (or identical) regular polygonal tiles. 

Only three types of regular polygon can tiled the plan: equilateral triangle, 
square or regular hexagon (see Fig. 1). Among these possibilities, the simplest 
is the tiling by squares: it requires the weakest number of tiled directions and 
satisfies to the purpose of this study. Accordingly, we choose the tiling by squares. 




Fig. 1. The three tilings of the plan by identical polygons: equilateral triangles, 
squares or regular hexagons. The simplest is the tiling by squares: only two tiled 
directions are required in this case (three tiled directions are required in the 
other cases). 



2.2 Simplest Tiliug aud Lowest Number of Squares 

In biological world, diversity is everywhere. Life, into many different forms, is 
diversity in action. Also, we must introduce some minimal diversity in the tiling 
process. In this goal, we choose to tile the plan with the lowest number of all 
different squares. This choice corresponds to a simple perfect squared square of 
lowest order (see Fig. 2). It is the most economic tiling possible and its mathe- 
matical form is uniqueness. As a result of graph and combinatorial theories, this 
economic tiling has been found by A.J.W. Duijvestijn [2]. 

To obtain the first generation square (the most external and integrative 
square of length 112), we add twenty different squares to the first square or 
germ (the most inside and little square of length 2). And so on, all along the 
successive generation squares... It is a russian doll process: to obtain a new gen- 
eration square, or new integron, we add 20 new squares to the former generation 
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square, or former integron. The properties of these mathematical integrons are 
analogous to the properties of the integrons defined by F. Jacob for the living 
systems [3]. 




Fig. 2. An Artificial Life integron: the simplest tiling of a square by the lowest 
number of all different squares. 



3 A Set of Analogies Between the Simplest Tiling of the 
Ideal Mathematical Plan and the Universal Genetic 
Code 

The simplest tiling, or integron, showed Fig. 2, is an asymmetrical figure. This 
figure marks the four directions of the plan: Up (U), Down (D), Left (L) and 
Right (R): i.e. a four letters alphabet in relation to the four sides of the asym- 
metrical figure. It exists also a twenty letters alphabet: the set of the twenty new 
squares added at each new generation. Between these two alphabets (the first 
of four letters and the last of twenty letters), it exists an intermediate alphabet. 
The sixty-four letters of this intermediate alphabet are the sixty-four elementary 
segments of the simplest tiling. 

With the same number of alphabets (3) and the same numbers of letters in 
these alphabets (4, 64 and 20), the structure of the universal genetic code (see 
Fig. 3) is similar to the structure of the simplest tiling. The first alphabet of the 
genetic code: the four nucleotides Uracil (U), Cytosine (C), Adenine (A) and 
Guanine (G). The intermediate alphabet: the sixty-four codons. And the last 
alphabet: the twenty amino acids (to make the proteins). 
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Fig. 3. The universal genetic code for the translation of the 64 codons into the 
20 amino acids. A codon is a mRNA word of three letters among four mRNA 
bases or nucleotides (U (Uracil), C (Cytosine), A (Adenine) and G (Guanine)): 
first (1st), second (2d) and third (3d) letters. 



Now, we define a set of “at least - at most” rules, or existence and asso- 
ciation rules, to specify the relationships between the 64 elementary segments 
and the 20 squares. 

Each square (among the 20 squares) is bounded by /3 elementary peripheral 
segments: 4 ^ ^ 7 (see Fig. 2). Existence or “at least” rule: to exist, each 

square (by analogy, each amino acid), must depend on one of its /? bounded 
segments (by analogy, of one codon). Association or “at most” rule: each 
square cannot depend on the totality of its /3 bounded segments. At the most, 
it depends on (/3 — 1) bounded segments: it is the only one manner to establish 
some minimal association links within the unified simplest tiling structure. 

By application of these rules, we have found the relationships showed Fig. 4. 
This figure exhibits some remarkable analogies with the genetic code. For exam- 
ple, the analogies that exist between the squares of length 35, 17 and 25 and the 
amino acids arginine, serine and leucine. 

There are only three squares (35, 17 and 25) to have the highest value of 
P {Pmax = 7). At the most, each of these three squares depends on 6 bounded 
segments {{/3max — 1) = 6). In the genetic code, this value (6) corresponds to 
the highest number of codons that code to one amino acid. And there are only 
three amino acids in this case: arginine, serine and leucine... 

In the genetic code, arginine and serine are adjacents (i.e. in the same box 
of the AGX codons), and leucine is alone. In the simplest tiling, we have some 
similar relative positions: squares 35 and 17 are adjacents and the square 25 is 
alone... 
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Fig. 4. Each arrow indicates a relationship between one particular bounded seg- 
ment and one particular square. In the genetic code, it exists a similar relation- 
ship between one particular codon and one particular amino acid. The three 
outside arrows correspond to the three Stop codons. 



4 Conclusions 

4.1 How to Explain the Set of Analogies Between the Simplest 
Tiling of the Ideal Mathematical Plan and the Universal 
Genetic Code? 

The results obtained and their precision lead to ask the question of a cause-to- 
effect relationship between the structure of the simplest tiling and of the genetic 
code. 

How to explain the links between the mathematical elements of the 2D sim- 
plest tiling (bounded segments and squares of an unified mathematical form 
that is independent of all matter substrates) and the particular 3D molecular 
substrates of the genetic code (mRNA codons and amino acids)? 

An answer can be found in a set of two dual graphs^ that is the mathematical 
basis of the simplest tiling: a first Up-Down graph^ links the horizontal opposite 
Up and Down sides of the Fig. 2, a second Left-Right graph links the vertical 
opposite Left and Right sides of the Fig. 2. The first graph has 11 vertices 

^ More precisely, two dual 3-connected planar graphs. 

® Every horizontal line of the simplest tiling corresponds to a vertice of the Up-Down 
graph, and each square (between two successive horizontal lines) corresponds to an 
edge. Every vertical line of the simplest tiling corresponds to a vertice of the Left- 
Right graph. 
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and 22 edges. The second graph has 13 vertices and 22 edges. These graphs 
are isomorphic to convex polyhedra. Accordingly to the Euler formula^: the 
first polyhedron has 11 vertices and 13 faces® (10 triangles, one quadrilateral 
polygon, 2 pentagons); the second has 13 vertices and 11 faces® (4 triangles, 4 
quadrilateral polygons, 2 pentagons and one hexagon). 

In [4] , it is suggested that the first genetic coding system can be started from 
specific interactions between two complementary 3D molecular substrates. 

From an optimal space filling point-of-view (to favorize chemical interactions 
and to minimize energy) , we think that the two unknown 3D molecular substrates 
have the same structures that the two previous dual polyhedra, and we propose 
to research the bio-molecules that could correspond {new way of research). 



4.2 Some Propositions for New Ways in Artificial Life Research 

The simplest tiling of this study, or integron, shows the emergence of an hierarchy 

of components (complexification process) and may be a potential source of new 

ideas (or open ways) in Artificial Life research: 

— New rules for programming cellular automata... For instance, in a “particu- 
lar” array (i.e. the integron of the Fig. 4 and its successive generations) with 
a rule that uses the arrows (to specify the authorized displacements) and 
works differently in each of the 20 squares of the Artificial Life integron... 

— Which is the equivalent simplest tiling, or paving, of an other space? In other 
words, which is the equivalent genetic code of this other space? In spaces of 
dimension^ ^ 3 ? In non euclidean spaces? Life “as it could be”... Life is it 
an exclusive property of the euclidean space? 

— Cross fertilization between the Artificial Life integron and the arithmetical 
relator language [5] . Arithmetical relator can express the world in structural 
levels (a quadratic form express the adaptation of the natural system to its 
environment). Artificial Life integron is also expressed by a quadratic form. 
The integron (see Fig. 4) presents a set of particular symetries: we think that 
the arithmetical relator - associated to the semisimple Lie algebra - can be 
an useful tool to study these symetries. 
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Abstract. This paper suggests that in the context of autonomous agents 
and generation of intelligent behavior for snch agents, a more important 
focns should be held on the symbolic context that forms the basis of com- 
pnter programs. Basically, software agents are symbolic entities living in 
a symbolic world and this has an effect on how one should think abont 
designing frameworks for their evolution or learning. We will relate the 
problem of symbol gronnding to that of sensory information available 
to agents. We will then introdnce an experimental environment based 
on virtual worlds called EMuds, where both human and artificial agents 
can interact. Next, we show how it can be applied in the framework of 
multi-agent systems to address emergence based problems and report 
preliminary results. We then conclnde with some ongoing and future 
work. 

Keywords: autonomous agents, symbol grounding, virtual worlds. 
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2 Artificial Intelligence and the Grounding Problem 
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2.2 Embodiment 
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2.3 Sensory Information 
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( orl ) h r pi r n onn t t In t on n r t th t on o 
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hr tr thnthtn ronm nt. hrough th h r t r th pi r n th n 
1 rtu 11 n th n ronm nt 1 ng to hum no 1 nt r t on 

r n or mpl xplor th orl n nt r t th omput r ontroll 
r tur . 

3.2 What Is an EMud? 

u t n or olut onr .n u nntnoth u 
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1 m tl num r o rtu 1 orl tht n g n nntlztonor 

th ppl t on h t n t xp r m nt n ronm nt. 

o g r pt on o th ppl t on tru tur t n r to un r 
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” th ng ” ) n exits r ng Ink rom on lo u nto noth r llo ng 1 m nt 

to mo n th orl n n ng t topolog . 1 m nt om n m n orm 

t p 11 th r objects creatures ( utonomou g nt ) or player- characters 
(hum n g nt ). Not th t 1 m nt m t on 11 lo n h h th 

r 11 containers, lloth Imntnlo r r nnntl 
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th k 11 o mo ng p t oor ( x t ) t ng n purr ng. ou oul h 



th k 11 o op n ng oor p k ng up oth r 1 m nt n o on. h orl 

oul r nntlztonloth orm n n gur 1 . 

Wh t r m n to n ho th 1 m nt hoo to ppl th r k 11 . 

h ppl t on n orpor t ontrol tmtht n o t n ontrol 
un t on to n 1 m nt pro th t th un t on h n nt r p 1 o 

nt rpr t ng m g nt th n ronm nt to th 1 m nt n p 1 o 

n ng omm n m g to th 1 m nt. ontrol Igor thm mu t pro 

gr mm plug n n th m hum n pi r h to un r t n 

m g rom th u h n onn t ng to t t In t on. 

3.3 EMud Ageuts 

n n u g nt r th 1 m nt th t po k 11 n th Ion n 
tr n orm th orl ppl ng th k 11 . u g nt houl on r 

o t r g nt 20 . S n th u llo n ontrol m h n m to u 
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BEGIN LOCUS Library 
SIZE 200 

CONTENTS Table, Book1 , Book2, Cat, You 
EXITS Hallway, Livingroom 
END LOCUS Library 

BEGIN ELEMENT Cat 
SIZE 8 

SKILLS look, move, purr, eat 
CONTROL CatBehavlor 
END ELEMENT Cat 

BEGIN CONTAINER You 
SIZE 15 

WHEIGHT ALLOWANCE 15 
SKILLS look, move, get, drop, open, close 
CONTENTS Books, Glasses 
CONTROL ByTelnet 
END CONTAINER You 



Fig. 1. tp 1 u ntlztonl 



or n g nt r nt 1 1 o utonom n h .hr un m nt 1 

1 1 n t ngu h 26 . rom utom t g nt utom t t mpl ng 

th t th g nt h to t n ronm nt n n nflu n ton utur 

X t n through t on n th t n ronm nt ( pr r qu t or n g nt to 

utonomou 22 ) on to op r t on 11 n h or 11 utonomou g nt . 
Op r t on 1 utonom n th It or th g nt to op r t thout 

hum n nt r nt on n h or 1 utonom uppo th g nt p 1 o 
tr n orm ng t pr n pi o op r t on. Su h n g nt mu t 1 to xpr 

r n h or t n or n ronm nt h ng . h mo 1 o 

n utonomou g nt r gur 2. n th pr 1 m n r tok n group 

ng xp r m nt pr nt lo r 1 ng th op r t on 11 utonomou 

g nt ut p rt rom llo ng hum n ontroll g nt to nt r n u 
n urth r xp r m nt 11 m nl on rn th u 1 ng h or 11 

utonomou g nt . 

3.4 The EMud in Operation 

rom th t rm nolog n r pt on t 1 r th t t th nt r o th p 
pi t on 1 oor n t ng nt t th t mu t h n 1 th nt r t on t n 

1 m nt po ng k 11 n th orl . n t u to th p r t on o th 

on m k ng Igor thm n th n ronm nt th t h n 1 h ul r 

h h n th 1 t o 1 m nt qu nt 11 r qu t ng omm n rom h 
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Fig. 2. n g nt r h t tur 
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3.5 Implementation Objectives 
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ontrol Igor thm or n u 1 or popul t on on to k p 1 rg p rt o 



xt 


n 1 1 to th 


ppl 


t on 


n 




0 r 


11 g 


m or nt purpo . 


h 


r fl 


t our nt r t 


0 


ng 


1 


to 


tn 


r nt 


ontrol Igor thm 


th n 


th 


m n ronm 


nt 


n 1 


0 


n 


on 


ph 


to 1 to ttr 


t th 


1 rg 


num r 0 t 


t 


t r 


1 


1 


rom th 


nt rn 


t h n t rt tu 


ng 


th 


t 0 hum n 


. nt 


r t on 


th 


omput 


r pt 


1 1 (not th t th 


t 


r 


on 0 th u 






1 




t In t to 


u p .un r. h on port 4000 


h n 


not 


un r r on 


n 


m 


1 to h 


tr 


)• h 


t 0 t p r p rt 


0 


tr n 


t on rom th 


tu 


0 


pt t on th n 


pur 1 


rtu 1 n m 1 


orl 



to orl h r om r 1 orl prmtrh to tkn nto ount 
th ontrol t m . 

4 EMuds as Experiment Environments 

u n tur 11 pro n ronm nt or th 1 1 n m t xp r m nt 

u h th rt 1 nt pro 1 m 4 or W 1 on oo xp r m nt 2 nt 

u to th r hn o un t on 1 1 1 1 or th r pt on o rtu 1 

orl n g nt t on th r n th r mor u t to ppro h mult g nt 

t m on omp rt 1 1. Sn gnt r mpl m nt plug n nt t 

r n n p n nt ontrol Igor thm t po 1 to rr out omp r t 
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xp r m nt th nt r t ng g nt ho ontrol Igor thm r o r 11 
r nt n tur . Wh n u th popul t on o g nt th po 1 1 op n 

n p r p t or th tu o oop r t on n omp 1 1 on. 

W urth r nt to 1 to on ront rt 1 g nt n th r n ronm nt 
th r 1 orl nflu n n th n on u ng th g m 1 k n tur 
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rom th nt rn t. h rupt on o r gul r p tt rn th n n n ronm nt 
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ontrol Igor thm u 

4.1 Preliminary Experiment 
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th xp r m nt n g nt n rr t mo t on tok n t n g n t m . Wh n 
mo ng th g nt r n oml 1 t on o th our 11 r t on th n 
qu 1 h n to go n h r t on. h onl n or n orm t on 1 1 to 

th g nt h r th num r o tok n n th 11 h t n ng n n th 
pr n or not o tok n n h o n n ntor . On n th p up r ult 

o th xp r m nt u ng on to ourt n g nt n th orl on gur 4 ) . h 

r ult r g n r t u ng gr ont n ng t n tok n . or 




Fig. 4. ) p up num r o g nt ) ) t p 1 xp r m nt ( g nt ) 



h num r o g nt hun r xp r m nt r run u ng r nt n t 1 

on gur t on th r ult ho n ng n r g o r th hun r 

on n th pro 1 m n ol th 1 n r to up r 1 n r p up u ng 
t o to X g nt . rom th n on mor g nt t n to o r ro th n ronm nt 
rupt ng th r group ng p tt rn xh t h n r g nt r n ol . o 

umm r z oop r t on m rg n th mult g nt t m h n th num r o 

g nt r ng rom t o to ght. On gur 4 ) n ) pi th o r 11 

r ult o hun r xp r m nt u ng g nt . gur 4 ) plot th num r 

o t r t on n to h th t k n h xp r m nt. gur 4 ) ho 
th tr ut on o xp r m nt r u th num rotrton. h rg 

num rotpu 4 thmx mum n m n mum ng 1 1 n 13 

r p t 1 un rl n ng th t th t u o th utonom o th g nt 
n though kno th go 1 11 r h nnot kno ho or h n. 

h or g n o th r group ng pro 1ml n th tu o oop r t on th n 

mo 1 ro ot g nt h r th un t on ng o hoi group p n on th 

n u 1 omp t n o h ro ot 11 th r nt r t on . m rg nt prop 
rt ouh tm rnlz n6n o port ng th m h or 
to th h p r mo 1 ro ot 1 . h nt g o u h n ppro h n oil 
t ro ot th t t 1 m n t th n or glo 1 po t on ng t m or o 

up r or mon tor ng th t k o tok n . h lo t on o th t k ont n ng 
11 th tok n r ult o th g nt nt r t on . 
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4.2 Further Experiments 

W r urr ntl mpl m nt ng mor ompl t n ronm nt h r t o t p 

0 h or 11 utonomou g nt o x t h ng r nt k 11 . n th x 

p r m nt g nt h un up r 1 rn ng p It n r r r or 

oil t ng tok n . On t p o g nt llo to p kup tok n h 1 th oth r 
n onl t 1 th m rom g nt Ir rr ng om . rom th tup 

xp t to o r ggr h or m rg ng n th th ng g nt hi th 

norm 1 oil t ng g nt 11 mo 1 1 k 1 lop p k up n run tr t g . h 

ontrol Igor thm or th g nt 11 m o 1 rn ng 1 r t m 
( S) 10 24 g ng th m oth m ol on t on t on rul m h n m n 

1 rn ng through r n or m nt n rul r t on. h nt r t on o t o 1 rn ng 

1 r t m 11 g n t on o ho rul ol h n on ront 

th n m 1 n ronm nt. 




Fig. 5. th S mo 1 o n g nt 
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ont nu 11 lu t th r n or m nt 1 rn ng Igor thm h h r r or 
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tr ngth n (r p. k n ) th 1 r or th n xt t m th t k p rt n n 
u t on. t r pr t rm n num r o t m t p th rul or t m 

g n r t n rul rom th tt t ol rul n r pi un t on n th rul 



5 Conclusion 
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nto ngl tmnml n u to omp n t or th kn o h 
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t m. On th on h n top on t m r non mo t m ho 
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th m n pul t n th r o n m ol un r 11 not r r rom g ng 
m n ng to o j t out th r orl ( . n th ph 1 orl ) n p r 
through n or . h g nt rtu 1 orl t ng It r through h h th 

ph 1 orl r h m ol 11 n mm tr to th t th h h 

hum n n r h nto th tr t orl through th r h gh 1 1 ogn t 

un t on on ph 1 pro 

W h th n pro to u 1 th u ppl t on pro ng r h rtu 1 

orl n ronm nt or rt 1 g nt to ol n. Our m n go 1 no 
to g n ull 1 xp r m nt t k ng nt g o th r t m ol to 

1 m nt n th orl 11 n lu ng hum n ontroll g nt h r ng 

th m n ronm nt. O nt r t oul 1 o to xt n th u orl 
po It mpl m nt ng tr ut r on u ng oor n t on 1 ngu g 

u h S 1 o th u ppl t on llo ng g nt to mo t n 
omput r r fl t ng m ol olog 1 n h . 
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Abstract. In artificial world research, the environments as themselves are rarely 
studied. Both in multi - agents systems and artificial life domains, the dominant 
work turns around the active entities, the agents. The aim of this paper is to 
propose a theoretical approach of an environment model. It must allow to deal 
with the different existing environment types but also to add a creative touch by 
introducing the concepts of meta - environment and multi - environments. This 
model defines some evolutionary schemes for an environment, and the different 
ways of interaction between them. 



Keywords: adaptive environments, environment model, environment 

interactions, multi - environments systems. 



Introduction 

John Holland quotes in one of its first papers, [1], « The study of adaptation involves 
the study of both the adaptive systems and its environment. In general terms, it is a 
study of how systems can generate procedures enabling them to adjust efficiently to 
their environments ». However, since 1962, most of the research work are focused 
mainly on the adaptive systems, rather than on their environment. This paper is a first 
step into the task of filling this gap, and proposes a theoretical study of an 
environment model. 

Even if most of the works on adaptive systems integrate the idea of a dynamic 
environment, generally, it only has a weak level of dynamism and it doesn’t really 
have a truly evolutionary process. That is why it seams important to consider the 
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environment not as a dynamic support where agents populations, [2], or animats, [3], 
evolve, but as an entity by itself. This entity will be able to develop particular 
behavior models, but also a non - fixed structure. These environment behaviors 
should be able to evolve following the different interactions with its objects and 
agents. 

Nowadays, most of the systems include an environment containing agents and 
objects. But, couldn’t a system integrate more than one environment ? In such a case, 
it is necessary to define, from on side, the different interaction laws that can exist 
between the environments and, from the other, a meta - environment being able to 
manage this laws considering each environment as an agent. 

The purpose of this paper is then to define an environment model which integrates 
all of these ideas. This model must allow to classify and to order the environments, 
giving by this way the possibility to determine transition and evolution stages 
between the different environment classes. 

The task of the first part is to establish this classification in a hierarchical way. 
Then, the second part presents the concept of meta - environment and the different 
interaction types that a multi - environments system can have. The next part concerns 
the definition of a protocol for the environments evolution and finally, the last part 
shows an example of a multi - environments system using the model previously 
defined in this paper. 



1. Environment Model 

This model lays, from one part, on the definition of a taxonomy of environment 
classes, and, from the other, on the use of the characteristics and parameters of each 
of these classes. From an object classes point of view, and also for simplicity of use, 
the chosen hierarchy is a simple one, i. e. each object have one and only one father. 

Taking into account the important use of some artificial life techniques such as 
genetic algorithms, [4], and classifier systems, [5], a genotypic description of the 
model classes is presented to conclude this part 



1.1. Taxonomy and Classes Characteristics 

This taxonomy defines five main classes. Two of them are abstract ones and the other 
three are concrete ones. The Fig. 1 shows this hierarchy. 

The abstract environment class is the root class of the hierarchy. It has some 
parameters, which are common for all its sub - classes : 

- The environment size, denoted S. This parameter determines if the environment is 
infinite or finite, and in this last case, S contains a positive integer value, which the 
unit depends on the environment type. 

- The set of the environment objects, O. This set can be decomposed in some parts : 
the active objects set, i.e. the agents, the inactive objects set and the sub - 
environments. 
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- The set of interaction laws between the different environment elements, I. This 
laws concern the interactions between agents and objects, agents and agents, but 
also the interactions between the environment and its components. 




Fig. 1. : Environments taxonomy. 



- The set of evolution laws of the environment, E. This set determines the rules 
followed by the environment to evolve in time, and also the transition functions 
which allow it to pass from one stage to another. 

The quadruplet <S, O, I, E> defines then an abstract environment. 

The situated environment class is generally used in the simulations concerning the 
multi - agents systems, and the different problems in artificial life. An environment is 
called situated if it is constructed from a space having a metric, and if its objects and 
agents have a position in relation to this space. The parameters of this environment 
type are : 

- The structure or architecture of the environment, A. Different structure types exist, 
as a toroidal space, square, cylindrical or spherical. 

- The environment dimension, D. The environment can have one, two, three or n 
dimensions. 

- The environment metric, M. This metric defines the measure units and the scale to 
define the objects coordinates within. 

- The set of the space management properties, P. This set is formed by some 
properties as the projection properties, from a dimension n to a dimension n-1, for 
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example, or even more, the rotation properties, useful for the visualization 
processes. 

In this kind of environment, the agents may have some particular properties, 
related to their situated aspect, their coordinates, their movement coefficient - or their 
speed - representing their capacity to move. 

A situated environment is then defined by the set <S,0,I,E,A,D,M,P>. 

The non - situated environment, N, is the class that can be used to define an 
environment in a ruled - based system. An environment is called non - situated if the 
concepts of space and position into space don’t exist in it. In this kind of environment, 
rules can be considered as the objects and meta - rules can be the agents. This class 
doesn't have any specific parameter, due to the fact that it is defined to be 
distinguished from the situated environment class. 

The discreet environment, D, follows the principle enounced by Bergson, [6] : 
« The intelligence only represents clearly to itself the discontinues ». An environment 
is called discreet if is it cut up in a finite number of elementary parts, the cells, each 
one having a fixed number of neighbors. The parameters of this type of environment 
are : 

- The cells structure, L. These may have different forms : square, hexagon, or 
triangle in a two - dimensional environment. 

- The integer environment metric. Mi. The environment being cut up in elementary 
parts, the computation of coordinates and moves are done in the integer set, or in 
one of its sub - sets. 

The continuous environment class, C, is the complementary class to class D. An 
environment is called continuous if, for every two points of it, there is at least one 
point between them. Even if, from a computing point of view, the idea of a strict 
continuity is impossible nowadays, a continuous environment can be built using a real 
metric, Mr, based on a real vectorial system. This kind of metric is the only parameter 
of this class. 



1.2. Genotypic Approach 

All the previous parameters can be coded to form a string, genome or chromosome. 
The elements of this genome, the genes, define then the different environment 
properties. Once this coding has been done, it is possible to apply a particular version 
of Genetic Algorithm (GA), used in artificial life area. 

This kind of algorithm would be near to the BEA, Bacterial Evolution Algorithm, 
presented by Chisato Numaoka in [7]. In opposition to most of GAs, having a large 
number of strings, this one considers entities, the bacteroids, owning only one 
chromosome. The bacteroids behavior is then completely determined from this unique 
chromosome. The evolution phenomena is performed just when the bacteroid, 
weaken', inserts in its organism the chromosomes of its neighbors. It chooses a 
chromosome in this set and, eventually, makes a mutation on this new chromosome. 

In the same way, environments following this approach have only one active 
string. This chromosome, which determines alone the environment genotype, defines 



' The bacteroid have an energy level. 
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the environment itself, its phenotypic representation, and its behavior. The 
chromosome fitness is then defined according to some parameters like : 

- The number of objects and agents in the environment. 

- The agents efficiency, which can also be measured as a fitness. 

- The utility of the environment laws. 

However, as a difference with the bacteroid case, the environment keeps the old 
chromosomes in a kind of memory base. This memory is then used in the evolution 
process to eventually return to an older, but more adapted, state. 

After the description of an isolated environment, a description of the possible 
relationships between different environments is presented in the next part. 



2. Meta - Environment Notion 

When the different elements of a system doesn’t act and interact with the same time 
scales, it can be useful to decompose it in a set of sub - systems. Effectively, it is not 
always coherent to consider in the same way the events produced in a micro - second 
scale and others produced in a year scale. The temporal windows of the evolution are 
completely different and cannot be managed with the same rules. 

At the environment level, this decomposition leads to the constitution of a multi - 
environments system. However, to take into account the different interactions that can 
be performed between the environments, it is necessary to establish the concept of 
meta - environment. This one doesn’t act as a supervisor but as a service manager, 
giving its services to its sub - environments, avoiding them some tasks. And even 
more, this approach allows a modular development, creating a hierarchy of 
environments and sub - environments, managed by a unique meta - environment. 



2.1. Characteristics 

A meta - environment is a kind of specialized abstract environment, having the 
following properties : 

- An infinite size. 

- A set of objects, exclusively environments. 

- A set of laws for the interaction between its sub - environments. 

- A set of internal evolution rules. 



2.2. Global and Local Meta - Environments 

It is important to point out the fact that a parallel can be established between the 
environment / agent relation and the meta - environment / environment relation. In 
the second case, the environment acts as an agent in relation to its meta - 
environment. 

Following this idea, an environment can contain other environments as 
components. In this case, the environment containing some sub - environments can 
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be considered also as a meta - environment. So, two types of meta - environment 
exist in this model : 

- Global meta - environment : the root of the multi - environments system 
hierarchy. Its properties are those defined in the previous section. 

- Local meta - environment : a node in the multi - environments system hierarchy. It 
is a component of another meta - environment, a global or a local one. It contains 
as a part of its objects set, a set of sub-environments, which can be considered as 
agents in relation to it. 



2.3. Inter - Environments Interactions Types 

A large number of interactions between environments can appear. This section 
describes some possible ones: 

- Environment dependence. An environment is completely dependant from another 
and can only perform its own actions according to its master environment. 

- Transfer. An environment can transfer either agents, or objects, or laws to another 
environment. 

- Communication. An environment can establish a communication with another 
environment in order to ask for, or to offer, either some information or a transfer. 

- Union. Two environments, both having almost identical elements, can join in order 
to form an unique environment, composed by the set of all the elements of both 
original environments^. 

The environment evolution can be caused, as in the next example, by its 
components, but also as a consequence of the evolution process of the other 
environments in its system. In this case, the environment evolution will be caused by 
a state change, controlled by its transition functions. Its corresponding meta - 
environment will detect this change and will inform the other environments. For 
example, if some environments share some resources, and one of them disappears, the 
meta - environment will transmit this information to the others to allow them to adapt 
to the new existing conditions. 

The next part describes an example of an environment evolution caused by its 
agents. 



3. Environment Evolution 

In the evolution process, an environment can have different states. An environment 
has an initial state just after its creation, a set of transitory states during its life period 
and a final state, the one before its death. All this states are linked together by 
transition functions. This environment evolution concept is important for the meta - 
environment, who needs to know the situation of its sub - environments in order to 
make the necessary changes for all of them, but also to create a new one or to destroy 
an existing one. 



^ An environment could also have the possibility to be cut, to give two distinct environments. 
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3.1. Evolution Types 

In the frame of the situated environments, these can be confronted to different kinds 

of evolution. This section presents the laws that allow them to evolve in different 

ways : 

- Structural evolution, the environment deeply modifies its structure. For example, 
lets have E, a discreet situated environment. If E is a two dimensions environment, 
defined by a 15*15 grid then, after a structural evolution, E could increase the grid 
size to 16*16, or decrease it to 14*14. This type of evolution can also occur on the 
dimension parameter, so a two dimensions one can evolve to a three dimensions 
one. 

- Behavioral evolution, the environment internal laws are modified. Considering the 
previous example environment, E, if it has some resources as a part of its objects 
set, an internal law can allow it to control the quantity of renewed resources at each 
step. A behavioral mutation will change this kind of criteria, increasing or 
decreasing it, in order to encourage or not the expansion of the agents feeding with 
this resource. 

- Component evolution, the environment acquires or destroys types of objects that 
may compose it. In this way, the environment E, evolving by this principle, could 
define a new kind of resource, more appropriated to one type of agents. 

- Geographic evolution, the environment creates or destroys some instances of its 
composing objects. In the environment E, after a geographic evolution, E would be 
able to create, on some cells^, new objects, new resources or new obstacles. This 
evolution type, where instances of objects are created or destroyed, must be 
distinguished from the previous one, where types of objects are created or 
destroyed. 



3.2. Evolution Example 

This evolution example uses a two layers neurons network to determine the mutation 
of the environment genome. This kind of evolution fits to the interaction between the 
environment and its agents. Then, some specials environment interaction effectors, 
represented in the network by input neurons, are affected to each agent. At each gene 
of the environment genome fits also threshold functions, causing a gene mutation 
after their overall activation. These threshold functions are defined by the output 
neurons of the network. As shown in Fig. 2, each output neuron is connected to each 
input neuron. 

When some agents activate the same environment interaction effector, the signals 
are transmitted to the corresponding output neuron. After that, if the function 
threshold is reached, a mutation of the corresponding gene will occur. This mutation 
may be interpreted as a demand of an environment modification from its agents. If a 
sufficient quantity of agents makes the demand, then, as a answer, the environment 
will adapt and transform itself, following the agents exigency. 



^ If the environment is discreete, or at a position in space if it is continuous. 
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It is necessary to make the right distinction of the output neurons, which represent 
the activate functions, and the genes of the environment genome. In the following 
example, the environment E is a discreet one, defined by a 10*10 cells grid. Then, a 
gene codes the environment size, however, two outputs neurons will be associated to 
this gene. The first one represents an increase of the environment size, if it is 
activated, while the second one represents a decrease of this value. Then, if a 
sufficiently important agents group demands to the environment, by the means of the 
neurons network, to increase its size, it could be able to undergo a mutation of its 
corresponding size gene. The environment will mutate then to a size of 1 1*1 1 cells. 



Agent 






Output 

neuron 



' Input 
'I neuron 

Set of associated 
fonctions to the 
environment genome 



Fig. 2. : Two - layers neurons network. 



4. Multi - Environments System Example 

In [8], Nils Ferrand quotes some possible characteristics of a multi - agents system, 

applied to spatial planning : « 

- Multiple scales : in space (from close neighborhood, to the nation or even 
continent), in time (from an instant for the perceived feelings of negotiators, to the 
centuries for the evolution of large scale ecosystems), and in organizations (from 
one - to - one interaction, to the transnational organizations) ; 

- Multiple actors : except in some very local and specific situations (like a 
landowner planning some limited works on its own territory, with no visual or 
ecological impact), spatial planning implies many actors, with different interests, 
social and organizational position, spatial attachment, and personal qualities ; 

- Multiple objectives : environmental, political, personal, economical, etc. 
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- Multiple modes : integrative (toward consensus through cooperation) or 

distributive (toward compromise through negotiation) ; 

- Multiple criteria : ecology, economy, landscape, agriculture, laws and regulations, 
culture, society, ...» 

The number of characteristics, non - exhaustive, denotes the difficulty of 
conceiving a complex multi - agents system using a unique environment and 
controlling a unique time unit. So, the necessity of building a multi - environments 
system appears, each of them having its own time and evolution criteria. 

In a more precise example, which the aim is to simulate a ground area with its 
different components, the evolution time scales are very variable. If the simulation 
takes into account the underground level, the ground level and the atmosphere, it’s 
clear that each of these elements and their internal elements doesn’t evolve at the 
same speed. In this way, the life time of an ant is shorter than the anthill life time, 
which is shorter than the tree life time. However, the relationships between them are 
non - negligible and must lead to a co - evolution of each element. These distinctions 
induce then the creation of many environments having interactions properties to allow 
this CO - evolution. 

The Fig. 3 shows an environment taxonomy allowing to model the 
anthill evolution, according to some possible sub - environments. 




Fig. 3. : Example of multi - environments system. 
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Focusing only on the development of two sub - environments, one corresponding 
to the ground definition and its properties [9] and the other, modeling an anthill [10], 
the following characteristics are considered : 

- Ground type (prairies, agricultural lands, forests, ...), water exchange (rain, 
evaporation, humidity level), carbon and nitrogen flow, temperature, plants 
decomposition (divided itself in some parts, following a chemical and physical 
protection index of the organic mater), plants production, ground textures (sand, 
clay, granite), ... 

- Ants types (queens, workers, soldiers), cocoons and larvae with different states, 
ants cadavers, multiple ants nutrition forms, ... 

Then, some interactions laws between these environments are indispensable to 
cause a CO - evolution of the different components. This laws concerns : 

- The substances transfers, an environment creates food for another environment 
agents, and these agents produce some rejects that allows the evolution of the first 
one. 

- The geographic aspect, the anthill being an environment geographically placed at 
the ground level, as shown in the Fig. 3. 

According to this decomposition principles it is possible to integrate, in a modular 
and progressive way, the other environment types to a developed system. 



Conclusions 

As it is shown in the example described in section 4, this environment model allows, 
from a practical point of view, to develop progressively a complex multi - agents 
system, defining and conceiving the sub - environments one by one. This model 
proposes different kinds of evolution for an environment. In this way, the agents 
evolution and the environment evolution leads to a co - evolving system. 

In order to simulate artificial ecologies, the environment model described in this 
paper, can be combined to a food - web model, like this presented by Lindgren and 
Nordahl in [11]. These authors show different existing food links between many types 
of animals and vegetables, but they also put forward some special links : « Not only 
do plants and trees provide food for the herbivore, they also provide shelter from sun 
and wind, escape routes and places to hide from predators, and branches and twigs to 
make nests and squats from ». These types of interactions prove the difficulty to 
conceive a realistic model of a natural ecosystem, but the concepts brought in this 
paper should help to develop modular platforms. 

Practically, the componential model, developed by Marc Lhuillier in [12], 
combined to this environment model would offer the possibility to construct multi - 
agents generic platforms. Those will tie together the notions of flexibility and 
dynamism of the componential model to the environmental evolution idea, described 
in this paper. In the future, a first platform of this kind will be developed, integrating 
the concepts of environmental evolution and meta - environment. Agents of this 
platform will be built with the macro - mutation concept, described in [13], allowing 
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an agent to undergo a deep modification of its genotypic structure, according to 
environmental conditions. 
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Abstract. AReVi (in French, Atelier de Realite Virtuelle) is a dis- 
tributed virtual reality toolkit. Its kernel (a group of C-I--I- classes) makes 
it possible to create cooperative and distributed virtual reality applica- 
tions by minimizing the programming effort. AReVi is built around a 
dynamic multiagent language: oRis. At any time, this language allows 
to stop the ongoing session, to add new entities (known or not when 
starting the session), to modify an entity or an entire entity family be- 
havior. More generally, oRis gives the user the ability to interact with the 
agents by directly using their language, thus offering a way of immersion 
through the language. 

Keywords: Virtual Reality, Software platform, 3D simulation. Multiagent 
Systems, Dynamic Languages 
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oRis m thod trigg rs th sso i t d ++ m thod 

oRis is o n d to oth r 1 ngu g s r Ir dy 1 to 11 Prolog r di- 

t s 22 or uzzy rul s insid n oRis rogr m to r t r tiv nd ognitiv 

g nts 

oRis w s su ssfully us d in H tions s v ri d s im g ro ssing 2 
immunology 3 d t ro ssing n tworks 4 or ins t o ul tions 16 

Integration to AReVi. h nks to th oRis ^ ++ ou ling AReVi off rs 

th us r r d fin d oRis 1 ss s (n tiv m thods whi h orr s ond to th r - 

viously r s nt d ++ 1 ss s (S t 3 2 ArEntity ArScene ArViewer t 

igur 3 r r s nts th di gr m of th m in r d fin d 1 ss s his di - 

gr m is th x t r fl tion of th ++ 1 ss s ( ig 2 
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Fig. 3. di gr m of th oRis r d fin d 1 ss s for R i 



Not 11 th 1 m nts m ni ul t d y AReVi r g nts not only th “ hys- 
i 1” Ar Entity o j ts from th univ rs ut Iso th stru tur s n d d for th 
simul tion s tting h m in dv nt g of this ro h is th t it is sy y 
d riv tion to giv th m h vior or x m 1 origin lly th ArScene g nt is 
only list of g nts whi h r visu liz d wh n w t hing th s n ith oRis it 
is ossi 1 to r t ProxyScene d riv d from n ArScene roviding h v- 
ior through main ( ) m thod or x m 1 this h vior my th utom ti 

int gr tion to th s n of ny virtu loj t(ny gntd riv d from Ar Entity 
nt ring th on rn d zon n th s m w y n AReVi vi w r (ArViewer h s 
ss to th fr m r t n v ry sily r t vi w r whos h vior is 

to r du th g n r t d im g qu lity (wir t if th r t om s too low 

n th n d t th r nd ring 1 v 1 to th ow r of th gr hi ngin 

4 Examples 

n this s tion w illustr t th R i multi g nt nd dyn mi iti s 

irstly w r s nt th mod ling of sim 1 m h nism ontinu with th 
r s nt tion of multi g nt v rsion of th P tri n tworks in lly w show 

V ry sim 1 w y to ou 1 oth r vious univ rs s th nks to th ov rd finition 

of m thods of inst n 

4.1 Trains of Gear Wheels 

wish to mod 1 tr in of g r wh Is h ro 1 m h r is not to r liz 

hysi 1 mod ling ( ont t tw n th ogs ut fun tion 1 mod ling two 

g rs in ont t rot t in th o osit dir tion th ngul r s dr tio ing 

id nti 1 to th r dius r tio 

h rolmmy t kldon gnrl oint of vi w th tr in h s four 

g rs g r 1 rot t s lo kwis g r 2 nti- lo kwis t 

tmy t kldon lo Iwy g rifisnoj t( ssiv t knows its 

su ssor nd its r d ssor in th m h nism t n 11 d ( 11 of m thods 

y its r d ssor P P indi t s to if how mu h it h s just rot t d s w 11 s its 
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h r t risti s hus E is su os d to d du its own rot tion nd tr nsmit it to 

its su ssor th mov m nt ro g t s st y st in lly w r t n ngin 

g nt (it is th only g nt of this xml whos h vior (main m thod is to 
rot t t only r m ins to “ onn t” th ngin to th first g r to st rt th lin 

n th d stru tor od from th r 1 ss w sk th kill d g r to dis onn t 

from its r d ssor hus wh n x uting if g r is kill d y th us r th r 
will no mist k indu d (th r vious g r no long r tr nsmits mov m nts 
nd th following g rs will utom ti lly sto sin no ody sk for th m 
ff tiv ly g t syst m whos h vior is onsist nt with its r 1-lif quiv 1 nt 
h r vious xml im 1 m nts ssiv g rs whi h r o j ts nt not 

g nts n ord r to g t mor r listi univ rs w n rovid th m with 
h vior 

— S rutiniz th univ rs 

— hn grifisdttdlos y (th ddition of its r dins nd th E 

r dins = th dist n tw n th ntiti s 

• onn tion to E 

utom ti lly st rt rot ting if if w s rot ting 

— f w r onn t d to g r nd if th dist n from it om s too im or- 
t nt 

• is onn tion 

utom ti lly sto rot ting 

hus now y moving th g rs with n int rf ri h r 1 (mous or Is w 
r 1 to modify th m h nism wh n this on is moving (s ig 4 




Fig. 4. 



rs n dyn mi lly nt in or t k n off th m h nism 
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4.2 Petri Networks 

h s end univ rs x m 1 w r s nt on rns th P tri n tworks su lly 
this kind of syst m do s not h v gr hi r r s nt tion ow v r w g v 
it on in ord r to visu liz its h vior (s ig 5 Not S v r 1 ro h s 
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Fig. 5. r hi r r s nt tion of P tri n twork 

xist for th rogr mming of P tri n tworks h n using oRis w h v s 1 t d 

n g nt’s ro h 1 s onn tions nd m rks r o j ts tr nsitions r 

g nts h y d id th ms Iv s wh n v r th y g t tiv t d it olvo t97 21 

4.3 Gears and Petri Network Coupling 

Now th n xt st onsists in r grou ing oth r vious x m 1 s in ord r for 

th P tri n twork to t s ontroll r for th g rs whi h thus will t s n 

o r tiv rt 

oth univ rs s w r uilt s r t ly hr for th y don’t know h oth r 
h ou ling is r liz d in thr st s 

1 Lo ding of th g rs 

2 hil th g rs r fun tioning lo ding of th P tri n tworks oth univ rs s 

o xist in m mory nd h v th ir own dis 1 y windows 

3 rigg ring of th omm nd on on of th n twork 1 s on m rk ntr n 

w st rt on th ngin ou 1 d to th g rs n n xit m rk w swit h off 
this ngin 

Point 3 is r liz d y dyn mi lly nt ring (without sto ing th syst m 
th od on ( ig 6 

his od ov rd fin s th ddition nd su r ssion m thods of PI 2 h 
m thods r not ov rd fin d on th 1 ss 1 v 1 (thus for 11 th inst n s ut 
only on rti ul r inst n 1 v 1 whi h th n do ts h vior slightly diff r nt 
from th oth rs 
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Overdefinition of a method "add” of the instance 2 from the "Place" class 




void Place. 2::add(agent mark) 



if (!_marks) 



{ 

Engine. l->wakeUp() ; 



We start on the engine when the place 
contains at least one mark 



Place::add(mark) ; 

} 



Call of the method "add" ftom the "Place" class 
(common to all instances) 



int Place.2::del(void) 

{ 

agent mark = Place: :del() ; 
if (!_marks) 



Engme.l->sleep() ; switch off the engine when piace 2 is empty 

return (mark) ; 

} 



Fig. 6. sso i tion of h vior to 1 of th P tri n twork 

Not during th ov rd finition of n inst n m thod it is Iw ys ossi 1 to 

11 m thod of 1 ss (for xml 11 of Place: :add() in 

Place. 2: :add() his v ry sim ly Hows to dd noth r tion to th d f ult 

h vior ro os d y th 1 ss 

5 Conclusion and Perspectives 

h AReVi 1 tform uilt round th oRis 1 ngu g is d v lo ing 1 tform 
for virtu 1 r lity li tions t h s n su ssfully us d in s v r 1 industri 1 

roj ts ( rgonomi s study of t hni 1 uildings s d s 3 visu liz tion 

3 int r tiv r r s nt tion of distri t rototy ing of m nuf turing sys- 

t m 8 h xist n of dyn mi multi g nt 1 ngu g in th syst m m k s 
it ossi 1 for us to 

— r t modul r univ rs s; th modul rity is r liz d y th s n of g n r 1 

ontroll r nd thus y th us of 1 m nt ry “ ri ks” with th ir own go Is 

th g nts 

— int r t fr ly in th s univ rs s; th dyn mi h r t risti s of th oRis 
1 ngu g How us s us rs to t on th low st 1 v 1 on th ntiti s ( v ry- 
thing n g nt is 1 to do us r is 1 to do it too th n r lizing n 
imm rsion through th 1 ngu g 

h r vious v rsion of th 1 tform did not in lud th oRis 1 ngu g ut 
How d th r tion of univ rs distri ut d through th n twork 10 
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now work on distri ut d v rsion of AReVi / oRis flowing th r rtition of 
g nts nd so of univ rs s through th n twork h low 1 v 1 ommuni tions 
r m d th nks to v nd su ort th diff r nt mod s ( ni st ro d st 
ulti st n our roj t th hoi of this 1 ngu g is n lizing wh n t Ik- 
ing out rform n s (-20 ut it off rs som v ry int r sting d v lo m nt 

f iliti s in t rms of sim li ity (multithr d rogr mming su ort d y this 
1 ngu g s f ty ( r - xisting grou of s f ommuni tion 1 ss s ( v- 

ns int gr tion of multim di ro ssing ( v di PI y rs high 1 v 1 

n twork xt nsion (s rvl ts R gl ts int ro r ility ( R nd on- 
n tion to th d t s s ( 
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Investigating the Complex with Virtual Soccer 
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Ibaraki, Japan 305 
{noda , ianf }@etl .go.jp 



Abstract. We describe Soccer Server, a network-based simulator of soc- 
cer that provides a virtual world enabling researchers to investigate the 
complex system of soccer play. We identify why soccer is such a suitable 
domain for the creation of this kind of virtual world, and assess how well 
Soccer Server performs its task. Soccer Server was used in August 1997 
to stage the simulation league of the first Robotic Soccer World Cup 
(RoboCup), held in Nagoya, Japan. This contest attracted 29 software 
soccer teams, designed by researchers from ten different countries. In 
1998, an updated version of Soccer Server will be used to stage the sec- 
ond RoboCup in France, coinciding with the real World Cup of football. 



1 Introduction 

ul s of so o sso i tion foot 11 w fo mul t in 1 63 in t K y 
t oot 11 sso i tion n t g m s sin om on of t most wi ly 

pi y in t wo 1 . popul ity of t g m is illust t y t following 
p ss g : 

ont y to popul li f t wo 1 ’s g t st spo ting v nt in t ms 
of p olong wo 1 wi int st is not t lympi ms. t it 
is t o 1 up of foot 11 w i lik t lympi s is 1 just on 
V y fou y s n is pi y out ov p io of two w ks o mo . 

nit t t s ost t is glo 1 sp t 1 in 1994 n w t 

sp t 1 it tu n out to . mpions ip g m tw n zil 

n t ly w s witn ss liv y ow of ov 100 000 p opl in t 
os owl in s n lifo ni n y ow of t 1 st illion 

on t 1 vision t wo 1 ov . 

quot t is s iption in ou int o u tion not simply us it illust t s 
t popul ity of so o us t pu li tion of t is p p oin i s wit 

n 9 t su sso to t min o 1 up. t w just s 
int st in t sou of t x pt not spo ts ook o n wsp p ut 

t op ning p g p of t op ning pt of ook on s i n : Would-be 

Worlds yon sti sti 97 . 
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sti’s pu pos in s i ing t foot 11 o 1 up is to int o u t no 
tion of complex systems, oot 11 is n x 11 nt x mpl in t is ont xt us 
it is not possi 1 to un st n t g m y n lysing just in ivi u 1 pi y s. 
t it is t int tions in t g ni ot tw n t pi y s t ms Iv s 
n twnt plysnt nvi onm nt t t t min t out om . is 

is t ss n of ompl x syst ms: t y sist n lysis y omposition. 

s sti’s li f t t t iv 1 of p pow ful wi sp om 

puting p ility ov t p st o so s p ovi us wit tool fo 

stu ying ompl x syst ms s ompl t ntiti s. sti ims If sugg sts t t su 
omput simul tions (vi tu 1 wo 1 s woul wo 1 s) pi y t ol of 1 o 
to i s fo ompl X syst ms w i in ont st to mo onv ntion 1 s i n 

1 o to i s How us to xplo info m tion inst of m tt sti 97 . t is 

to p ovi t is kin of 1 o to y fo s on ompl x syst ms t t w v 

V lop simul tion oft g m of so . is syst mo v n 1 s 

so m t to pi y tw n two t ms of pi y p og ms n si 

y n us to st g 29 t m ont st t t i st o oti o o 1 

up ( o o up) in N goy pn. ntispp w si o v 

1 ntifying in p ti ul w y so is su suit 1 oi s simul tion o 

m in. Iso t k to v In t t su ss of o v monst ting 

t t t p op ti s of fidelity simplicity clarity bias-free n tractahility 

tiv m su s fo ss ssing t is kin of simul tion. 

2 Soccer as a Domain for Modelling 

w m ny sons in t oi of so fo on simul to . niti lly 
of on s t w s t pp 1 t t so is fun n sily un stoo y 

1 g num s of p opl . o impo t nt t n t is ow v w s t signi nt 

11 ng o y t ying to fo m lis n un st n t om in. 

2.1 Soccer as a Complex System 

o illust t t i ulty of t so om in 1 t us tu n to t is ussion of 

ompl X syst ms t t w st t in t nt o u tion n to sti’s ook Would- 

be Worlds, n is ussing ow t o y of t ompl x mig t fo m lis sti 

int o u s num of t isti s s i s s t k y ompon nts 

of ompl X ptiv syst ms. i fly t s (1) m ium siz num of 
g nts t t (2) int llig nt n ptiv n (3) only v lo 1 info m tion. 
t s oul not too to onvin t t t t g m of so is 

V y goo t fo t s p op ti s. o s 22 g nts. is f 11s omfo t ly 

tw n sti’s X mpl s of syst ms wit 1 g num s of g nts (g 1 xi s 

w i V noug g nts to t t st tisti lly) n sm 11 num s of 
g nts ( onfli ts tw n two sup pow s). g nts in so Iso in 

t llig nt n ptiv wit t pi y s on t m st iving to p fo m w 11 
tog t n to out m noeuv t oppon nts. in lly t info m tion in so 
is 1 ly limit s t pi y s n only s in t i tion t y f ing 




Investigating the Complex with Virtual Soccer 243 




Fig. 1. in ow im g of o v 



n V 1 ommuni tion is mp ot y ist n n y t impo t n 
of on ling int ntions f om t oppon nts. 

us un st n ing so is signi nt 11 ng : it s t t isti s 

of ompl X syst m fo w i s sti mp sis s t xists no nt 

m t m ti It o y sti 97 . ow v s w 11 s ing goo x mpl of 
ompl X syst m not impo t nt f to Iso influ n on oi of so : 

t f si ility of oust u ting simul tion. x min t is qu stion low. 

2.2 The Nature of Play 

n of t fo most t sks in t sign of simul tion is t tion of model 

of t om in in qu stion; it is t impl m nt tion of t ul s n p in ipl s 
of t mo It tpou st simul tion. n t is sp t w foun t t so 

w s V y m n 1 om in simply y vi tu of ing g m . 

n X 11 nt stu y of t n tu of pi y is giv n y uizing uizing 0 . 

It oug p im ily int st in t signi n of pi y s ultu 1 p 
nom non som of t m in t isti s of pi y i nti y uizing 
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1 V nt to ou t m .0 inst n t st p op ti s of pi y i nti 
y uizing t t it is volunt y (o s uizing s ys 1 y to o is no 
long pi y ) n t t pi y is not o in y o 1 lif . s p op ti s 
1 y sugg st som kin of mo lling p o ss. uizing t n not s t t 11 

pi y is limit in lo lity n u tion not f tu t tmksply 

p omising n i t fo onv ni nt simul tion. 

u t g n 1 p op ti s i nti y uizing t t 11 pi y s in 
ing ulsnttplyn pt nt nsmitt . p s n of 
ul s m ns t t p t of t wo k of mo lling is 1 y on . n in im 
pi m nting o v w took t simpl pp o of sing ou mo 1 

p im ily on just t ul s of t g m . m in of t o v 

mo 1 w s t n fo mul t to Ilow m ximum s op fo t inv stig tion of 
ow t ul s of so n tu Ily follow in t st w y; t t is ow to 

t t m t t n pi y w 11. uizing ’s n 1 p op ty of p t ility n 

t nsmitt ility t n in i t s ow ou mo 1 n us s n i 1 t st 

fo flowing s s to inv stig t n is uss t p op ti s of t om in. 

2.3 The Type of Model Represented by Soccer Server 

s w sugg st ov o V mo Is t ul s of t g m of so in 

w y t t Hows t syst m to us s t st fo s on t n tu of 
t om in. int ntion is t t i nt so pi y ont ol Igo it ms n 
t st g inst ot to is ov w i st ong . us o v 

n vi w s p i tiv mo loft 1 tiv st ngt soft s Igo it ms. 
g wit sti 97 g 2 t t t st n fo most t st t t mo 1 

must p ss is t t it must p ovi onvin ing nsw s to t qu stions w put to 

it . Not t ong t t t qu stions w w nt to nsw wit o v 

not just w i Igo it ms p fo m tt nt Iso why t tt Igo it ms 

sup io . is is ons qu n of vi wing t syst m s t st fo s ; 

t ultim t go 1 is to v n t t o y of ompl x syst ms. 

s oul Iso not t t It ong t is p p on nt t s on t vi tu 1 

wo 1 tyo vt 00 up ton n m nts t ms Iv s Iso in 

In omp titions fo t ms of 1 o ots. n ou vi w t is link is v y im 

po t nt sin t m ny suits t t s ow t t 1 wo 1 xp im nts 

n p o u suits t t oul not xp t f om simul tion Ion {e.g. s 
ompson 97 ooks 6). it tis v tsi t nit us mov on to t k 
t il look t ou impl m nt tion of mo 1 of so 

3 Soccer Server Itself 

o V n 1 s so m t to pi y tw n two t ms of pi y 

p og ms (possi ly impl m nt in i nt p og mming syst ms). m t 
using o V is ont oil using fo m of li nt s v ommuni tion. 

o V p ovi s vi tu 1 so 1 (su s t on w p s nt in 

igu 1) n simul t s t mov m nts of pi y s n 11. li nt p og m 
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Soccer Server 




Fig. 2. V vi w of o V 



n p ovi t in’ of pi y y onn ting to t o v vi 

omput n two k (using / so k t) n sp ifying tions fo t t pi y 

to y out. n tu n t li nt iv s info m tion f om t pi y ’s s nso s. 

igu 2 giv s n ov vi w of ow to v ommuni t s wit 

li nts. t m in mo ul s in t o v its If : 

1. A field simulator module. is t s t si vi tu 1 wo 1 of t 

so 1 n 1 ul t s t mov m nts of o j ts king fo ollisions. 

2. A referee module. is nsu st tt ulsoft gm follow . 

3. A message-board module. is m n g s t ommuni tion tw n t 

li nt p og ms. 

low w giv n ov vi w of oft s mo ul s. mo t il s iption 

of t o V n foun in No et al 9 . Iso 1 g oil tion of 

It m t i 1 in lu ing sou s n full m nu Is is m int in 1 1 o 

V om p g : http ; //ci . etl . go . jp/~noda/soccer/server. 

3.1 Simulator Module: The Basic Virtual World 

so 1 n 11 o j ts on it 2 im nsion 1. is m ns t t unlik 

in 1 foot 11 t 11 nnot ki k in t i n t pi y s nnot m k 

us of skills su s ing o voll ying. siz of t 1 m su in t 
notion 1 int n 1 quiv 1 nt of m t s is 10 m x 6 m. n t 1 t o 

j ts t t mov (t pi y s n t 11) n o j ts t t st tion y (fl gs 

lin s n go Is). st tion yoj tss v slnm kst tli nts nm k 

us of to t min t i pi y ’s position o i nt tion o sp n i tion of 

mov m nt. li nts not giv n info m tion on t solut positions of t i 
own pi y s ut only on t 1 tiv lo tion of ot o j ts. us t x 

o j ts impo t nt f n points fo t li nt p og ms’ 1 ul tions. 

11 ommuni tion tw n t s v n li nt is in t fo m of 

st ings. fo li nts n liz in ny p og mming nvi onm nt on 

ny it tu t t s t f iliti s of / so k ts n st ing m nipul 

tion. p oto ol of t ommuni tion onsists of: 
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— Control commands: ss g s s nt f om li nt to t s v to ont ol 

t li nt’s pi y . si omm n s turn dash n kick, om 

muni tion is on u t t oug t say omm n n p ivil g go li 

li nt n Iso tt mpt to catch t 11. sense_body omm n p o 
vi s f k on li nt st tus su s st min n sp n change_view 
s 1 ts t o tw n qu lity of visu 1 t n f qu n y of up t . 

— Sensor information: ss g s s nt f om t s v to li nt si ing 

t u nt st t of t g m f om t vi wpoint of t li nt’s pi y . 

two typ s of info m tion visu 1 (see) n u ito y (hear) info m tion. 

o V is is t simul tion of ontinuous tim . us ot t 

ont ol omm n s n t s nso info m tion p o ss wit in f m wo k 
of simul to st ps’. 1 ngt of t y 1 tw n t p o ssing st ps fo t 
ont ol omm n s is 100ms w s t 1 ngt of t st p y 1 fo t s nso 

info m tion is t min y t most nt change_view omm n issu y 

li nt (w is uss t timing of t s st ps in mo t il in §4). Not t t 

11 pi y s V i nti 1 iliti s (st ngt n u y of ki king st min 
s nsing) so t t t nti i n in p fo m n of t ms iv s f om t 

tiv us of t ont ol omm n s n s nso info m tion n sp i lly 

f om t ility to p o u oil o tiv viou tw n multipl li nts. 
s n 1 f tu w n invok wit t -coach option t s v p ovi s 

n xt so k t fo p ivil g li nt ( 11 o li nt) t t s t ility 

to i t 11 sp ts of t g m . o li nt n mov 11 o j ts i t 

t f mo ul to m k isions n nnoun m ss g s to 11 li nts. is 
f ility is xt m ly us ful fo tuning n ugging li nt p og ms w i usu 
lly involv s p t t sting of t vio s of t li nts in m ny situ tions. 

n xt nsion ing onsi fo fu t v sions of o v is mo i 

V sion of t -coach option t t Hows t ms to in lu tw 1ft li nt t t 
s glo 1 vi w of t g m n n on u t si lin o ing u ing pi y y 

s outing st t gi o t ti 1 vi to pi y s. 

3.2 Referee Module: Playing by the Rules 

f mo ul monito st gm n sin Iso gul t s t flow 

of pi y y m king num of isions s on t ul s of t g m ( n 

noun ing go Is t ow ins on ki ks n go 1 ki ks n tim k ping), n 

t st o o up tou n m nt t f m no isions out fouls, 
n w V sion of o v ow v int o u s n o si ul ing to t 

possi 1 ju gm nts oft f . m ining fouls lik o st u tion’ i ult 
to ju g utom ti lly s t y on n pi y s’ int ntions. s v t fo 

Iso in lu s n int f llowing um n us to inst u 1 1 f to w 
f ki k. in su isions out f i pi y tu lly 11 fo signi nt un 
st n ing of t g m futu v sions of o v m y in lu it 

so k t fo f li nts t us n ou ging s on utom ti f ing. 
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3.3 Message Board Module: Communication Protocol 

m ss g o mo ul is sponsi 1 fo m n ging t ommuni tion 

tw n t pi y s. si p in ipl of o v is t t li nt s oul 

ont ol just on pi y n t t ommuni tion tw n in ivi u 1 li nts 

s oul only i out vi t say n hear p oto ols int on in §3.1. 

n say omm n is issu t m ss g is o st to all li nts imm 
i t ly s u ito y info m tion. n ly v sions of o v t ms oul 

us t is ommuni tion to si st p t lo In tu of info m tion in t g m 

y fo X mpl ving pl y o st is own lo tion n int p t 
tion of t g m t tim st p. not possi 1 st t gy w s to us on 

li nt in t m {e.g. pt in’) to i t n info m 11 t ot s. n t 

u nt o V ow V not 11 t ot li nts gu nt to 

11 t o st info m tion sin t is m ximum ng of ommuni 
tion of Om n ny pl y n only on m ss g f om t m u ing 
two simul tion y 1 s. Iso t 1 ngt of t m ss g its If is st i t 

us tiv n i nt us of t say omm n is n ou g imp ting 

info m tion only w n it is us ful n tim ly. 

Not t t t s V n onn t wit up to 22 li nts ut t t to f ilit t 

t sting it is possi 1 fo singl p og m to ont ol multipl pl y s ( y st lis 

ing multipl so k t onn tions). n omp titiv situ tion su s o o up 
su p og ms only p mitt on t un st n ing t t t ont ol of 
pl y is s p t logi lly. n p ti it is Iso not t ni lly i ult to 

t t s V n st lis int li nt ommuni tion outsi t s v so 

t on li nt on pl y ’ ul is tiv ly g ntl m ns’ g m nt. 



3.4 Uncertainty 



n 0 


to 


fl t t n tu 


of t 1 wo 1 t sv intousv 


ious 


typ s of 


un 


t inty into t 


simul tion s follows: 




— Nois 




to t mov 


m nt of 0 j ts. mount of t is nois in 


s s 


wit 


t 


sp of t 0 , 


,i t. 




— Nois 




to omm n 


p m t s. m 11 n om num s 


to 



t p m t s of omm n s s nt f om li nts. 

— Limit omm n x ution. s v x ut s only on omm n fo 

pl y in simul tion yl. lintpogm ngtf k 

on ow m ny omm n s its pl y s x ut vi us of t sense_body 

omm n (successful x ution must of ou s monito y t li nts 
t ms Iv s; fo inst n dash s no t if pl y ’s p t is lo k ). 

— n X t s using. fut nojtt Iss lilt info m tion 
tu n out it f om t s v . 

p s n of t s un t inti s nfo s t impo t n of o ust v 

io n of t tiv monito ing of t out om s of pl y ’s tions. 
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4 Assessing the Soccer Server 

ow n w m ningfully ss ss t o v ? n §2.3 w sugg st t t 

t m in t st of goo mo 1 is w t it p ovi s nsw s to t qu stions w 

sk of it. us sin w onst ut o v to nit inv stig tion 

of so w oul simply x min t qu lity of t on fusions 1 n out 
g nt viou in t om in. ow v mo fun m nt 1 vi w is Iso 
possi 1 . o inst n sti 97 g 17 176 summ is s s v 1 p op ti s 
t t n us to ss ss mo Is n simul tions: 

— Fidelity. mo I’s ility to p s nt lity to t g n ss y to 

nsw t qu stions fo w i t mo 1 w s onst u t ; 

— Simplicity. 1 v 1 of ompl t n ss of t mo 1 in t ms of ow sm 11 

t mo 1 is in t ings lik num of v i Is ompl xity of int onn tions 

mong su syst ms n num of ad hoc ypot s s ssum ; 

— Clarity. s wit w i on n un st n t mo Int p 

i tions/ xpl n tions it o s fo 1 wo 1 p nom n . 

— Bias-free. g to w i t mo 1 is f ofp ju i soft mo 1 

ving not ing to o wit t pu po t fo us o pu pos of t mo 1. 

— Tractability. 1 v 1 of omputing sou s n to o t in t p 

i tions n /o xpl n tions o y t mo 1. 

Not t t t s qu liti s mo su tl t n simply ss ssing w t 

mo 1 f it fully ptu s 11 sp ts of t p nom non it p s nts. L t us 

X min t suits of v lu ting o v g inst of t it i . 

4.1 Fidelity 

n of t most impo t nt onsi tions u ing t v lopm nt of o 

V wst Ivlof st tion fo p s nting t li nt omm n s n 

t s nso info m tion. n possi ility w s low 1 v 1 p ysi 1 s iption 

fo X mpl flowing pow v lu s fo iv moto s to sp i . ow v it 

w s f It t t su p s nt tion woul on nt t us s’ tt ntion too mu 

on t tu 1 ont ol of pi y s’ tions 1 g ting t u inv stig tion of t 

multi g nt n tu of t m pi y to t 1 v 1 of s on y o j tiv . u t 
it is i ult to sign low 1 v 1 s iption t t is not impli itly s on 

sp i notion of o ot w ; fo x mpl ont ol of sp y iv moto s 
is i s tow s p ysi 1 impl m nt tion t t us s w Is. n t ot 

n mo st t p s nt tion m y using t ti 1 omm n s su s 

pass-ball-to n block-shoot-course woul pou gminwi t 
1 wo 1 n tu of so om sosu ninwi t v lopm nt of 

so t niqu s not y t lis y um n pi y s om s p o 1 m ti . us 

ou p s nt tion using si ont ol omm n s su s turn dash n 

kick is omp omis . o m k goo us of t v il 1 omm ns li nts 

will n to t kl ot t p o 1 ms of ont ol in n in ompl t info m tion 

yn mi nvi onm nt n Iso t st w y to om in t o ts of multipl 
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pi y s. us w li V t t o V i v s ou go 1 of p ovi ing 

simpl t st n wit signi nt 1 wo 1 p op ti s. 

oi of st tion 1 v 1 is Iso 1 v nt to fu t qu stion t t oft n 

o upi us u ing t sign of o v : w t t simul to s oul 

sign to mo 1 um n so pi y o o ot so pi y. is qu stion 
s m ny f ts ut in g n It solution opt in t o v is 

to f it ful to um n so w n v possi 1 . justi tions fo t is 
in lu t soning t t 1 wo 1 so is mo imm i t ly un stoo 

( n mo tt tiv ) to t su 1 o s v n t t o ot t nology n 

ng w s t n tu of um n so pi y s is 1 g ly oust nt. Iso 

t los t simul tion to 1 wo 1 so t mo lik ly t t knowl g 
quisition f om um n xp tis {e.g. s nk 97 ) will i tly ppli 1 . 

foust i nstwn Iso nt mo 1 p s nt 

y t s V most not ly t 2 im nsion 1 n tu of t simul tion. Iso 
t simul tion p m t s tun to m k t s v s us ful s possi 1 fo 

t V lu tion of omp ting li nt syst ms t t n to i tly fl t lity 

(fo X mpl t wi t of t go Is is 14.64m ou 1 t siz of o in y go Is 

us s o ing is mo i ult w n t 11 nnot ki k in t i ) . is 

qu stion of w t um n so is ing i tly simul t is Iso impo t nt 
in t ont xt oft ov 11 o o up 11 ng w i w is uss fu t low. 

4.2 Simplicity 

ultim t go 1 of o o up s si in Kit no et al 97 is to v lop 

0 ot so t m t t n 1 1 zil wo 1 up t m . in t is go 1 is 

w 11 yon u nt t nologi s t o o up initi tiv ns s i s of w 11 

i t su go Is t t i V 1 in t s o t n mi t m. mpl m nting 

li nts fo o V is on of t s 11 ng s. o j tiv of o o up is 

t t st Ivlof sop isti tion of xisting t nology imp ov s t simpli ity 

of t simul tion s oul It pp op i t ly. 

n of t p im y on ns out simpli ity is t timing of t simul tion 

y 1 s t t gov n t p o ssing of omm n s n s nso info m tion. s n 
so info m tion iv y li nts is i ut t tions simpl . is is n 
gum nt fo m king t tim tw n info m tion up t s long t n t tim 

tw n tion x ution (z. e. ving n tion p o ssing ylt t is sot 

t n t info m tion s nsing y 1 ). ft info m tion s nsing y 1 om s 

too long t ility to t to t oppon nts’ tions om s mp . n 

t ot n if t info m tion s nsing y 1 om s too s o t t n ts 
of tt mpting to 1 n n p i t t oppon nts’ tions iminis 

u nt 1 ngt s of t s tim st ps omp omis int n to st ik In 

tw n t s two onfli ting go Is. 

4.3 Clarity 

1 g num ofs svl yus o v(wvl y 

m ntion t t t st o o up ont st f tu 29 t ms f om t n i nt 
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ount is). is is vi n t t t wo kings of o v n sily un 

stoo . n onsi ing t 1 ity’ of t syst m t oug w Iso v to t k 
into ount t s wit w i on n un st n t 1 ssons 1 n t oug 
t on s of su s . o w int st in qu stions lik t suits 

V m g out of t t ms pou yt stooup ont sts? o 

som xt nt t nsw to t is typ of qu stion is p n nt on t o ts of 
t o V us s t ms Iv s. ow v t o o up initi tiv qui s 11 

t t ms nt ing ny tou n m nt to w it p p si ing t i pp o 

sppssowt tint o igin 1 ont sts t p og ms t t p fo m 

w 11 w simpl o syst ms wit littl 1 ning ility. o x mpl t 

t 0 0 up’96 tou n m nt 1 in p n in Nov m 1996 t winning 

t mwst gltswi ss nti lly li on oust ining pi y to 
st y wit in sm Up t min of t pit n to oos tw n 

sm 11 num of op ssing i tions w n t y oul t 11. 

n t o o up 97 tou n m nt 1 in ugust 1997 on t ot n t 
winning t ms w mo sop isti t . winn s f om um ol t niv 

sity us s s soning n g nt o i nt p og mming t unn s up 

us info m nt 1 ning n t t i pi t m us n xpli it mo 1 of 

t mwo k s on joint int ntions. 

s w 11 s monst ting t 1 ity of o v it s oul point 

out t t som tim s t us s of t syst m n s ow Imost t opposit : t t 

t syst m w s not tu lly fully un stoo y t sign s t ms Iv s. is 

pp ns w n ugs foun in t simul tion t t n xploit to t 

V nt g of t li nt p og ms. n x mpl of su ug w s n o in t 

impl m nt tion of st min . n lly t st min of pi y s s w n 

t pi y issu s dash omm n s t us limiting t pi y ’s ility to m k 

fu t mov m nts. ow v it w s foun y som us s t t y using negative 

p m t s in t is omm n pi y ’s st min oul in s ! 

fu t fl w w s foun wit t v sion of o v us fo t 

0 0 up 97 ont st in w i p o 1 m wit t n ling of simul tion y 1 s 

som tim s Ilow pi y s to ki k t 11 t tim s in v y qui k su ssion 
t us imp ting t tim s t no m 1 m ximum’ sp . ow v it is in t 
spi it of t o o up ont st t t su ugs po t to t sign s so 

t t 11 us s wo king f om n v n footing. is spi it is Iso n illust tion 
of t i s f n tu of o V . 

4.4 Bias-Free 

It oug V lop sol ly t t 1 t ot ni 1 L o to y o v 

s n t su st nti lly f om t opinions n sugg stions of m ny us s. n 
p ti ul t is m iling list it to is ussing o o up on w i 
ons nsus is usu lly fo mo ifying t mo 1 p s nt y t 

simul tion (t is m iling list v g ov 200 m ss g s p mont tw n 

1997 n 199 ). o giv f 1 fo t sp of ng of t simul to 

is p ti 1 list of som of t f tu s ng sin o o up 97: t 
int o u tion of t sense_body omm n t ility to see t i tion t t 
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ot pi y s f ing t int o u tion of o si s t int o u tion of go li 

n mo lling of t impl m nt tion of st min . 11 t s ng s 

sign to in s p op ti s su s t simil ity of t simul tion to t 

um n g m t impo t n of mo lling t oppon nts’ pi y o t impo t n 
of mo lling t t mwo k ing p o u y li nt’s own t m m t s. 

ut vinoft isf ntuofo vistpsn 

of pu li li y of o (t o o up imul tion o iv lo t 

t http : //www. isi . edu/soar/galk/RoboCup/Libs/). s s n us t is 

0 to Ip u t V lopm nt tim of o o up li nts. om o is sp 

1 to p ti ul situ tions w s som is g n 1 noug to p ovi inf 

st u tu fo ompl t g nts (t o fo t winning li nts f om o o up 97 
is V il 1 ). n p ti ul t iv s ont in t li s li nt’ li y oil 

tion of low 1 V 1 fun tions int n to st t w y m ny t ils of t sks su 

s m int ining so k ts p sing t info m tion f om t s v n s n ing 
omm n s to t s v . 

4.5 Tractability 

o V s to simul t g m of so n ommuni t wit 

li nt p og m in 1 tim . syst m n un on o wo kst tion wit 
only mo st m mo y n p o sso sou s ( It oug typi lly us will 

Iso qui t 1 st on fu t omput to un t t ms of li nts). mo 

p o 1 m ti limit tion on o v in p ti is t p ity oft n two k 

onn ting t syst m to t li nts. 1 g lo on t n two k n 1 to 

ollisions t t p v nt li nt omm n s ing o v o info m tion 

ing tu n . n som t ms t o o up 97 foun it n ss y to 

li t t i softw to t p ti ul on itions foun t t tou n m nt 

sit . ow V it is i ult to p i t t x t ts of ng s in n two k 

on itions so li nt p og ms wit som fl xi ility in t i int p t tion of t 

st t of pi y n ou g . n g n 1 t is m y not n v s p op ty 

of t syst m; t n to un st n pt tion n to op wit only lo 1 
info m tion w two of t p im qu liti s of ompl xsyst ms i nti in §2.1. 

5 Conclusions: Towards a Theory of the Complex 

V is uss ow t g m of so is goo x mpl of ompl x 
syst m w 11 suit to t t sk of mo lling. Iso s i n ss ss ou 
impl m nt tion of o v 1 ifying t n tu oft mo 1 it p s nts. 

L t us los y noting t t t 11 ng s p s nt y t om in of 
so V ntly 1 to it ing p opos s nwstn polmfo 

s Kit no et al 97 . f ou s t notion of st n p o 1 m s 
long n iving fo fo ngin ing s . isto i lly fo x mpl t 
pt n of t u ing st’ u ing 0 fo us tt ntion on t mimi king 

of um n vio s t st of m in int llig n . o ntly ss s 

iv signi nt tt ntion g n ting impo t nt v n s in t t o y of 
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s Igo it ms n s ont ol s w 11 s motiv ting ognitiv stu i s into 

t w ys t t um nsppo t smpol ms {e.g. s L vinson et al 91 ). 

o (in p ti ul ont st to ss) is yn mi 1 tim multi g nt 

syst m wit in ompl t info m tion. s o v s vi tu 1 wo 1 t t 

p ovi s tool fo inv stig ting su ompl x om ins. t o s t is y p ovi ing 
p i tiv mo 1 of t st ngt s of softw Igo it ms fo ont oiling g nts 
in t s nvi onm nts. n t wo s of sti 97 o v is woul 

wo 1 t t s t p ity of s ving s 1 o to y wit in w i to 
t st t ypot s s out t p nom n it p s nts . t is on op t t fo 
ompl X syst ms su s so t o ts of s s using t 1 o to y of 

o V will tify t situ tion w t p s nt t s ms to no 

known m t m ti 1 st u tu s wit in w i w n omfo t ly ommo t 

s iption sti 97 g 214 . 
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Abstract. This paper presents Webots: a realistic mobile robot simu- 
lator allowing a straightforward transfer to real robots. The simulator 
currently support the Khepera mobile robot and a number of extension 
turrets. Both real and simulated robots can be programmed in C lan- 
guage using the same Khepera API, making the source code of a robot 
controller compatible between the simulator and the real robot. Sensor 
modelling for ID and 2D cameras as well as visualisation and environ- 
ment modelling are based upon the OpenGL 3D rendering library. A hie 
format based on an extension of VRML97, used to model the environ- 
ments and the robots, allows virtual robots to move autonomously on 
the Internet and enter the real world. Current applications include robot 
vision, artihcial life games, robot learning, etc. 
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2 Real Mobile Robots 
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5 Current Applications 
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5.2 Artificial Life Games and Robot Football 
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Abstract This paper presents the implementation of the panoramic, lin- 
ear, artificial retina EDI within the frame of the Webots mobile-robotics 
simulator. The structure and functioning of the simulation are presented. 
Realistic sensor readings and software compatibility are the two key con- 
cerns of the simulator. A realistic sensor output is obtained by modelling 
the physical processes underlying the sensors. Interface compatibility be- 
tween the real and simulated retina has been made possible through the 
use of the Khepera API. The simulator is validated by running a control 
algorithm developed on a real robot equipped with such a vision sensor 
and obtaining the same behavior. 



1 Introduction 
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n simul t h p o ot uipp ith su h tin . Th on lu in 
m ks is uss th p oj t its ppli tions n utu o k. 



2 The Panoramic, Linear, Artificial Retina EDI 

Th ^ p no mi lin ti i 1 tin is n n lo -VL vision hip 
si n y Oliv L n olt t M( uhtl itzln) int n to v lu- 
t th p iliti s o su h ioinspi s nso s. Th Mi op o sso L o to y 

(L M ) o th L is u ntly o kin on th ppli tion o su h s nso s to 

mo il o oti s n most p ti ul ly on h p o ot. 

Th tin h s lin y o 150 photo io s lyin on n th t 

ov s 240 s o vi . ut s its n m su sts th is mo th n 

simpl vision s nso p 1 o t ns u in li ht. s in most nim 1 tin s 

som simpl si n 1 p o ssin t k s pi in th vi inity o th photo t to s 
still in th n lo om in. This lo 1 p o ssin h s th to It in th 
in o m tion u in its n i th n p ovi in th n xt st ith mo 
1 V nt in o m tion. n s s 3 sistiv i s 11 n om in to 

^ EDI is a shorthand for ED084V2A, the CSEM’s device number. 
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pply lo p ss n p ss o hi hp ss It on th im . no -impuls - 

spons It n Iso ppli p o u in th iv tiv o th im . Th 
It output n ss y mi op o sso th nks to th int n 1 / 

onv t p s nt on th tin . n this y 64- y-1 v 1 im n us 

y th o ot o ny u th p o ssin . 

Th tin h s n mount on h p tu t kno ns no mi 

tu t to th ith th n ss y n ill y vi s n i s i uits. Th tu t 

h s n in lu s plu -in mo ul in th ots simul to . Th olio in 
s tion is uss s th simul tion o th no mi tu t n th tin . 




Figurel. Th h p no mi tu t in o po t s n ti i 1 lin 

tin . 



3 The Panoramic Turret on the Webots Simulator 

Th simul tion o th tin olio s its un tion 1 lo ks hi h omp is 

th opti s th photo t to s th It s n th i it 1 onv sion. h lo k 
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3.1 Optics 

Th st st p o th simul tion is to 1 ul t th in i nt li ht int nsity on 
h photo t to . ill onsi th pix 1 v lu s o th Op n L n in 
n in s th li ht int nsity llin on th pix 1 su . Th 1 tin h s 

sph i mi o n 1 ns th t o us th im on th photos nso s hi h li 

on th hip s s n in u 2. lly th in i nt li ht th t h s th sili on 

su shoul th s m s th in i nt li ht th t hits in on th sph i 
mi o . T kin this hypoth sis ill t y to o t in th in i nt li ht on this 
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Figure2. Th opti s o th tin 
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Figures. omposition o 240- s ylin i 1 vi into 2 fl t vi s. 
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V ntu lly th V ti 1 pix Is v to o t in t o 75 x 1 pix 1 v to s 

nthtovtos ontnt topou 150 pix 1 v to . 




Figure4. oj tion o o 1 in sph i 1 oo in t s on to fl t s n 



3.2 Photodetectors and Normalization Circuit 

Th im o t in in th p vious su s tion is onsi s m su o 

th in i nt li ht int nsity on h o th 150 photo t to s. Th im 
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olo h nn 1. tt mo liz tion oul hi v i th sp t 1 spons 
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n t is onsi s th photo u nt on h photo io . xt no - 
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3.3 Filtering Layer 
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V sion. Th o impuls - spons It is n t y su t tin th output 
om t o uni i tion 1 i usion n t o ks. t p o u s th iv tiv o th in- 
put im n it n us to t t sh p s in th s n . 

1- sistiv i usion n t o ks h v s lo -p ss It so xpon nti 1 im- 
puls spons in th sp ti 1 n sp ti 1- u n y om ins 

h{n) = ^ H{u) = I • 

sistiv i usion n t o ks p 11 1 lo lly- onn t syst ms th t p - 

o m omput tions on 1-tim sis. This m k s it i ult to t on 

sin 1 p o sso . o V sin th syst m is m out o sisto s th t is 

lin vi s it n sho n th t th output is th onvolution o th input 
si n 1 ith k n 1 o impuls spons o th o ms th t n s n in u 
5. 




Figures, mpuls spons s o th lo p ss (-) hi hp ss (--) n o (-.) It s. 



Th suit o onvolvin th analog [ ] y ith th sistiv n t o k 
impuls spons is sto in th filtered [ ] y. 

3.4 Digital Conversion 
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3.5 Software Interface: The Khepera API 
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V lop on ots simul t o ot nnot sily ppli on 1 o ot. 

To th t n h V v lop ommon int o th In simul t 

no mi tu t n h p o ot in th o m o n ppli tion o m 
nt th h p 5 . u h ommon int llo s th us to us 

th s m sou o ithout h vin to h n sin 1 omm on oth 
pi t o ms. T o i nt li i s on o th 1 o ot n on o th vi tu 1 

on p ovi ith th ots simul to . 

Th h p ont ins s v 1 un tions to 
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solution n th ion o int st. 

4 Experiments: A Robot Regatta 
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Obstacle AvoidanceBehavior 




Light Seeldng Behavior 



FigureG. Th n u 1 n t o k us in th tt xp im nt 
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o ot ont oil . mo tho ou h v li tion oul ui volvin th s m 
h vio on In simul t o ot. 
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Abstract 

We suggest in this article a new paradigm for the representation of data, which 
is best suited for the real-time visualization and sonorisation of complex 
systems, real or simulated. The basic idea lies in the use of the garden 
metaphor to represent the dynamic evolution of interacting and organizing 
entities. In this proposal, multiagent systems are used to map between given 
complex systems and their garden-like representation, which we call Data 
Gardens (DG). Once a satisfying mapping has been chosen, the evolution of 
these Data Gardens is then driven by the real-time arrival of data from the 
system to represent and by the endogenous reaction of the multiagent system, 
immersing the user within a visual and sonorous atmosphere from which he 
can gain an intuitive understanding of the system, without even focusing his 
attention on it. This can be applied to give life to virtual worlds by grounding 
them in reality using real world data. 



1 Introduction 

Let's imagine a virtual garden whose visual and sonorous aspects continuously change 
to reflect the passing of time and the evolution of weather conditions in a distant 
place. Looking at it or simply listening to its musical rhythm will make you feel just as 
if you where there, looking at your garden through the window. ‘It's raining cats and 
dogs. Better stay home! ’ Connected to real meteorological data, it really functions as a 
virtual window, opened on a distant reality. This is what the computer-art project 
called The Garden of Chances (GoC to make it short) [11] is all about. Beyond its 
artistic interest, we believe it to have very important implications for the 
representation of complex systems by means of visual and sonorous metaphors. 
Keeping a close watch on meteorological data in order to secure airplanes landings, 
monitoring the physical condition of a patient during surgical operations, observing 
Stock Market fluctuations so as to determine the best options to chose, are three 
examples of situations where decisions are subjected to the real-time understanding of 
complex systems, respectively physical, biological, and social or economical. Those 
representation and interpretation issues are transposable for artificial complex systems 
such as multiagent systems, for which adequate real-time representation may provide 
insight into the inner mechanisms of the system at the agent level, or topsight [10] 
over the functioning of the system as a whole. Visualization in Scientific Computing 
(ViSC) has proven very efficient to represent huge sets of data, by the use of statistical 
techniques to synthesize and class data in a hierarchical way, and extract relevant 
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attributes from those sets, before presenting them to the user (Fig. 1). But it has not 
been so successful when dealing with distributed and dynamic systems since it is 
based, among other things, on a delayed treatment of the data. 

The basic proposal is to consider any complex system one wish to represent as a 
metaphorical garden, the evolution of which reflects in real-time the evolution of the 
system. In this paradigm, the measures made on the system are not only stored, 
waiting for a further statistical processing, but they are also immediately transmitted to 
a Data Garden, a virtual ecosystem with the same global dynamics as the system to 
represent but with a stronger visual and sonorous appeal (Fig. 1). Indeed, the garden 
metaphor has the interesting property to be both very complex in its functioning, and 
still completely familiar to anybody, enabling a very fast and intuitive perception. 
Moreover, it doesn't require a sustained attention, since it relies for the most part on 
peripheral perception mechanisms, following the same principles as those that make 
us perceive weather conditions effortlessly. Finally, the Data Garden paradigm 
doesn't reduce the complexity of the system to represent but transform this complexity 
to integrate it into a meaningful environment, creating a visual and sonorous ambient 
atmosphere from which to gain a continuous understanding of the studied system. 

In section 2, we present the concepts of scientific visualization and we explain why 
they fail to satisfy the needs of complex systems representation. By contrast, we 
present in section 3 The Garden of Chances, an artistic project which succeeds in 
mapping numerical meteorological data in an abstract, yet meaningful, representation 
of the weather. We finally extend the principles developed with this project in section 
4, explaining the characteristics that Data Gardens should share in order to prove 
meaningful for complex systems representation, before concluding. 

2 Scientific vs. Artistic Visualization and Complex Systems 

Scientific visualization on the one hand is based on quantitative information display 

[22] , visually in most cases but also using different modalities [4]. Fig. 1 shows the 
classical iteration cycle of scientific visualization whereby experiments are 
undertaken, during which data are collected. Only afterwards are the data analyzed 
and visualized, which allows to draw conclusions about the experiment and design 
complementary experiments. The cycle then iterates. This is well fitted for a great 
number of applications but doesn't qualify for the representation of complex, dynamic 
and distributed phenomena. Painting on the other hand, considered as a system of 
colored graphical elements is inherently distributed and based on the organization of 
those elements. According to Kandinsky, “analysis reveals that there is a construction 
principle which is used by nature as well as by art; the different parts become alive in 
the whole. Put differently, the construction is indeed an organization” [15]. 
Furthermore, painters try, and sometimes succeed, in transmitting complex perceptive 
and emotional experiences to their spectators. They so establish what J. Wagensberg 

[23] calls a “communication of unintelligible complexities”, complexities that 
language and numbers cannot express since they cannot be formalized. 

We're now going to analyze in further details the reasons why scientific visualization 
concepts appear inadequate to us for the representation of dynamic and complex data. 
It appears in the classical taxonomy that a processing of the data is necessary in order 
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to extract from huge sets of data, a restricted number of attributes that best synthesize 
the nature of the data. To this purpose, a great number of statistical techniques and 
data analysis are available that we won't detail here [3]. The results are then presented 
using a number of standard representations such as histograms, pie or time charts and 
so on. New presentation models [19] are also developed that make the visualization 
easier by focusing on some specific aspects of the data depending on the context. An 
alternative to this general scheme is when the phenomenon has physical reality and 
may be visualized directly or using appropriate color scales. Physical numerical 
simulations make a large use of this techniques, and medical visualization is a rapidly 
expanding domain that also exploits the same principles. 
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Fig. 1. Classical (partial view from [6]) vs. Data Gardens' visualization taxonomy 

We propose to handle, with artistic visualization, phenomena which are both 
distributed and with spatial and temporal dynamicity. Our hypothesis, based on the 
analysis of scientific visualization techniques, is that such complex systems cannot be 
well represented using purely quantitative and objective means. This may be explained 
by the fact that we have an almost qualitative and subjective experience of such 
complex phenomena as meteorology, biological ecosystems, social groups, etc. Most 
of the knowledge that we have about those systems is derived from our everyday-life 
perceptions, which give us an intuitive grasp about such systems but which we don't 
know how to transcribe into numbers. 

3 The Garden of Chances 

We have explored with the computer-art The Garden of Chances an artistic alternative 
that is useful in making qualitative aspects visually or sonorously sensible in the 
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representation of complex systems. Fnrthermore, the distributed aspect of complex 
systems is integrated as the basis of the functioning of the project and we think it 
qnalifies as the first step in the representation of complex systems by means of colored 
and sonorous metaphors. 

3.1 The Artistic Paradigm 



The philosophy underlying this artistic work is to let the automatic generation of 
images be directed by a real time incoming of real world data. This has led to the 
development of a first computer artwork called Quel temps fait-il au Caplan? (What's 
the weather like in Caplan?). In this project, weather data coming in hourly from 
MeteoFrance stations were used to suggest the climatic atmosphere of a given spot 
(actually a small place in Britain) by means of color variations inside an almost fixed 
abstract image. To pnt it naively, rather warm tints were used when the temperature 
was high, dark tints when clouds appeared to be numerous, etc. In addition to 
meteorological parameters, the system also took astronomical ones (season and time 
of the day) into account, which eventually allowed very subtle variations. When 
fnnctioning continuonsly all year long, the animation makes the compnter screen 
become a kind of virtual window, giving access to a very strange world, both real and 
poetic. 




Fig. 2. The Garden of Chances 



The GoC (Fig. 2) is basically designed with the same principles, namely using real 
data for the creation of mixed worlds, imaginary landscapes anchored in real world. In 
addition to colors modnlations, the weather data are used to give life to a set of two- 
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dimensional shapes, so as to create a metaphorical representation of a real garden. 
Thus, each graphical creature is able to grow up like a plant, benefiting from the 
presence of light and rain, competing against similar or other hostile shapes, 
reproducing and dying like any living creature. By so doing, the goal is definitely not 
to produce accurate simulations of natural ecosystems nor realistic pictures of 
vegetation. The focus is rather put on enabling the artist to experiment with lots of 
different abstract worlds until he obtains some imaginary ecosystem fitting his 
aesthetic sensitivity. The graphical space doesn't have the passiveness of coordinate 
systems anymore; we rather consider it as an active principle giving birth to worlds, as 
the raw material from which everything is created. 

3.2 The Multiagent System 

In agreement with artistic requirements, the system has been implemented as a 
programmable platform, allowing the artist to undertake a true artistic research. 
Capitalizing on our experience with biological simulation systems [7], we designed it 
as a genuine vegetal simulation platform, supplying growth, reproduction, and 
interaction mechanisms similar to those observed in plants. Indeed, we believe the 
difference between metaphorical and simulated ecosystems only resides in the 
perspective adopted during the experimentation process. 



agents 




The core of the platform is a multiagent simulation system (see Fig. 3), representing 
plants as very simple reactive agents evolving in a simulated environment. Both the 
agents and the environment are characterized by sets of parameters that define their 
characteristics at any given time. The activity of the agents is defined as a number of 
behaviors which are programmable using a little scripting language. Those behaviors 
are handled by a scheduler which activates them whenever needed, either periodically 
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(each n simulation cycles) or upon reception of some particular events (those are 
related most of the time to changes in one or several parameters). Finally, agents will 
be represented on the screen by colored shapes, which won't have necessarily 
something to do with plants but may be freely designed by the artist. A given still 
image will thus be close to his painting work, while the dynamics of the whole system 
will more closely rely on the artificial side of the project, i.e. the simulation of natural 
processes of vegetal growth. 

3.3 Agents and Environment 

Parameters constitute the basis for the representation of both agents and the 
environment. Actually, six types of parameters have been defined in order to describe 
the simulated world and the incoming data flow. 

Agents are characterized by reserved, internal and external parameters as shown in 
Fig. 4. Reserved parameters are common for all agents whereas internal and external 
parameters may be defined specifically for each agent. Reserved parameters include 
information about age, speed, color, size, etc. Internal parameters describe the 
resources of the agent (water, glucose, etc. with the vegetal metaphor, or any other 
quantifiable resource). By contrast, external parameters represent any substance or 
information that the agent may propagate around him (chemical substances that plants 
release in the soil or the atmosphere, signals, etc.). 





Fig. 4. Agent's parameters 



Fig. 5. Environment's parameters 



The environment is characterized by local and global parameters as shown in Fig. 5. 
Local parameters correspond to variables whose value and evolution can be defined in 
a local way, i.e. for each square of the grid covering the environment (substances 
present in the soil, water or mineral materials for example). On the contrary, global 
parameters represent variables which have a nearly uniform action on the whole 
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environment (meteorological variables, real world data, etc.). Finally, each data 
variable within the incoming data flow is integrated as a data parameter. 

3.4 Behaviors 

A little scripting language based on Tcl has been developed in order to make the 
programming of behaviors easier to handle from a user's point of view. In this 
language, one may think about agents' and environment's parameters as local and 
global variables, and behaviors may be thought as procedures, either with local effects 
when associated to agents or with global effects when associated to the environment. 
Within behaviors, there are only four basic actions which can be combined together 
using conditional statements (if ... then ... else) and iteration loops (for ..., while ..., 
repeat ... until). These four actions are the modification of a parameter, the death or 
reproduction of the agent, and the playing of a sound file. For example, if one assumes 
that sun-shining conditions in a distant location are available in real-time in the data- 
flow and we wish to make a flower grow according to this parameter, one would have 
to express that some flower gains energy through photosynthesis if the sun is shining, 
loosing a part of this energy continuously due to its metabolism, and dying if it doesn't 
have any energy left. The programmed behavior would simply look like the following: 

// parameters declaration 
data param sun_shining 3 
internal_param flower energy 50 0 100 

// photosynthesis behavior declaration 
behavior flower photosynthesis 1 
// energy parameter modification 

param_set energy {$energy - 0.01 + $sun_shining * 0.1} 
// death if not enough energy 
goc_if {$ energy == 0} 

death 

sound flower_death.au 

end_if 

end_behavior 



This approach allows to describe how some parameters evolve (slow decrease of the 
energy of an agent to sustain its metabolism, etc.), to specify the interactions between 
the agents (chemical aggressions, etc.), between an agent and its environment (water 
drawn from the soil to feed a plant, etc.), and even between different levels of the 
environment (supply of underground waters by the rain, etc.). 

Behaviors are triggered by a scheduler, either with a fixed periodicity or whenever a 
given event occurs. An event is simply defined as a special kind of behavior which 
periodically evaluates a condition and positions the corresponding flag of the 
scheduler accordingly. In agreement with the vegetal model, different behaviors of a 
single agent may be triggered concurrently during a single timestep. A simulated plant 
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can thus simultaneously execute different operations such as drawing water from the 
soil, realizing photosynthesis, growing, releasing chemical substances, etc. 

3.4 So What? 

Set aside aesthetic and artistic discussions, the scientific evaluation of the quality of 
such a system proposing a metaphorical visualization of real data, would have to be 
done under two complementary points of view : 

• what can the spectator say about the data that were used to generate the 
pictures ? In other words, does the representation used for visualizing the data 
make sense for the spectator ? 

• what can the spectator say about the dynamics of the system that produced 
those data ? Does the representation succeeds in producing topsight ? 

Since the GoC has originally been developed in an artistic perspective, experimental 
protocols have yet to be designed in order to scientifically address these issues. Our 
personal experience with the system is that meteorological variations are easily 
detected, but the GoC would surely prove not so well adapted for the visualization of 
any kind of data, especially when complex systems with no physical reality are 
concerned. We feel however that this artistic approach to both complex systems and 
data visualization may provide us with new paradigms for the visualization of complex 
data. 

Without entering the details of painting creation, it should be underlined that painters 
are used to handling the problem of organizing distributed colored shapes in order to 
produce a given global effect. They are also used to evaluating and interpreting the 
produced result. Actually, they are used to communicating complex and subjective 
information by means of a visually distributed representation, and this expertise will 
be valuable in further development and experimentation steps. We're now going to 
formalize these intuitions to show how they should be integrated in a single effort to 
make complex interacting data accessible to direct visual perception, in what we have 
called Data Gardens. 

4 Towards Data Gardens 

The expression Data Gardens was coined in analogy with the artistic project and its 
garden metaphor. In fact. Data Ecosystems may be more appropriate, indicating any 
community of living and interacting organisms with graphical and/or sonic 
representation, whose individual survival is subjected to the real-time incoming of 
data to which they react, and to their interactions with other entities. Still, Data 
Gardens is strongly evocative and we will only use this expression in the remaining 
sections. 

Data Gardens are an alternative proposal to standard schemes of data visualization 
which we think would be most efficient when applied to the representation of complex 
systems for purposes of diagnostic or monitoring. We explained in section 2 why we 
thought those standard schemes to be inadequate in the context of complex systems. 
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and we presented with The Garden of Chances the outline of a possible alternative. So 
what are the basic features that Data Gardens should integrate in order to fulfill the 
requirement of adequately representing complex systems, that is providing the user 
with a global understanding of the functioning of the system. And how could this be 
achieved ? 

• As a first thing, DG should rely on a dynamic representation, necessary get a 
perceptive sensation of a system's dynamics and to easily detect discontinuities. 
This representation should be both visual and sonorous, vision being best fitted 
to the perception of spatial dynamics through a parallel treatment of 
information while audition is more adapted to the perception of temporal 
dynamics through sequential treatment of information. The aim is to limit the 
use of high-level cognitive processes, trying to take advantage of the 
“peripheral”, rather than active, perception capabilities of the user (many 
experiments in augmented reality, like in the MIT MediaLab's Ambient Room 
[12], rely on this approach). 

The graphical and sonorous aspects of The Garden of Chances have been 
designed so that the user can interpret the painting in the same way he could 
interpret his daily environment for extracting critical information. Animation 
on the other hand is an essential part of the GoC, and it carries out two 
different kinds of information via two different dynamics: a slow, 

homogeneous dynamics, intended to reorganize the whole environment during 
a long period of time, is used to represent the seasons. Within this dynamics, a 
few punctual and rapid animation of some agents or sets of agents are used to 
represent important short-term fluctuations of the data values. 

• When representing complex systems, complexity shouldn't be reduced a priori 
with synthetic indices and means, but should be directly integrated in the 
representation system. Therefore, Data Gardens should be based on a 
multiagent modeling (accurate or metaphoric) of the system to represent. But 
because complex data are generally just too complex to represent directly, 
distributed principles for organizing and synthesizing the represented data must 
be integrated, that decrease the perceptual load of the user without significantly 
altering the meaning of the data. 

In Data Gardens, incoming data influence both the activity and the evolution 
of the agents. The synthesis is then realized at two levels : in the individual 
evolution of each agent, and in the mutual interactions they engage in. The 
development of an agent, which is graphically translated by a modification of 
its shape or color, can be the consequence of the variations of different data, 
like the evolution of a plant within an ecosystem. 

• The representation must be metaphorical, that is, use cognitive categories 
aimed at being easily interpreted and managed by the user. This is a way of 
decreasing the complexity, mapping abstract data into a meaningful 
representation from which one can get instantaneous understanding. 
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In that respect, one of the most interesting aspects of Data Gardens is the 
“garden” or “ecosystem” metaphor, already developed in the GoC. Analyzed at 
the light of the first two points we developed, the garden metaphor has the 
interesting property to have natural significance to anybody, while being very 
complex in its functioning. This functioning relies on the interactions of three 
types of complex systems: physical (the weather), biological (vegetal and 
animal) and social (social animals such as ants). This results in various audio- 
visual dynamics which look very familiar, from the slow evolution of vegetal 
landscapes to the fast interactions of animal life, and continuously changing 
meteorological ambiances. 

The representation should be programmable, at a user's level. The user must be 
able to express subjective choices about the representation, either to make it 
more significant with regards to data visualization, or to make it more 
aesthetically pleasing. Perception is a mostly individual and subjective feature 
of human cognitive functioning. The representation should therefore fit the 
user's own subjective conceptions of how the data are best visualized. 
Moreover, different types of representation of the same data may reveal 
different aspects of this data. 

In the GoC, the whole simulation system is programmable allowing the user to 
filter the input data, define the behaviors of the agents to make them react to 
the data, and choose their graphical representation. In the process, the user will 
need to be guided by the system, which should propose sets of dynamics and 
graphical vocabularies. Our collaboration with a painter in the GoC project has 
proven very useful for that particular matter, and we intend to cooperate more 
closely with various artists in order to define basic instances of Data Gardens. 

The representation should be interactive to allow dynamic exploration and 
perturbation of the multiagent system. Once a general framework has been 
found efficient, it must be made possible for the user to refine the 
representation acting as a kind of gardener by adding, removing, moving, 
changing agents on the fly. The user should also be able to visualize the system 
from different points of view. If the default functioning of Data Gardens 
exploits peripheral perception, the user may also choose to focus his attention 
on a specific part of the representation, getting then more detailed information 
about this part. Finally, the user should be integrated as a particular agent of 
the system, with extended capabilities, that would enable him to experiment the 
reactions of the system when perturbed. 

In the GoC, the basis for these interactions is set. The user can make agents 
reproduce, die or move, and one can view or change the parameters of agents 
or of the environment. When associating specific behaviors to a “perturbing 
agent” and when moving it around with the mouse, one can also visualize how 
the system may spontaneously reorganize. 

The system should finally be evolutive, taking into account the interactions of 
the user in order to learn something of his tastes and evolve accordingly. The 
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reason is that a given user will most of the time be unable to express how he 
would like the data to be represented in the formalism of the system. But if he 
can't formalize it, he can point out aspects of the representation that he finds 
pleasing or some others that he dislikes, just as in the gardener metaphor. This 
last point is mostly prospective for now, since nothing like it exists in the GoC 
yet. 

As a result. Data Gardens should be designed as “meaning operators” between the 
flow of data and the user, who is supposed to identify and follow, in the evolution of 
the system, that of the outside world. However, it must be clear that they are not 
intended to replace the existing environments used to track and trace data (histograms, 
textual presentation, curves, etc.), which are still the only way to know the precise 
value of a variable. They are to be viewed as complementary tools that allow an 
instantaneous and natural perception of complex situations and propose a global 
perspective on them. The Garden of Chances is the first of such systems we built and 
it should now be studied in a systematic way, and with an experimental perspective, in 
order to develop operational design and evaluation methodologies. 

5 Conclusion 

We propose with Data Gardens hybrid environments that graphically represent 
information gathered in the real world for users likely to take decisions in this real 
world. The goal is to let a complex and dynamic system of numeric data become 
visually intelligible without catching the whole attention of the user. We explained 
how the application of DAI concepts, along with real-time visualization, allows to 
compensate for some of the deficiencies of more classical approaches. In particular, it 
enables to handle dependent data without a priori reducing the complexity of the data 
but only as the result of a dynamic hierarchisation and synthesis of the different pieces 
of the data through the organization of a multiagent system. Furthermore, the 
representation is evolutive and adaptive, because of its dedicated endogenous 
evolutionary mechanisms, and also because of the user's actions. We presented with 
The Garden of Chances, both the artistic work that initiated the reflection on Data 
Gardens, and the first concrete application of this paradigm. But it is obvious that 
Data Gardens are not limited to given data nor graphical representation types, each 
application domain requiring however that the representation be adapted to specific 
cultures and representation habits. Further work will put the focus on systematizing 
this adaptation process, through the constitution of libraries proposing various 
dynamics, based on previous works in DAI, and various graphical and sonorous 
schemes, based on a collaboration with artists. The aim is ultimately to be able to 
interpret complex and interacting systems of data, almost as naturally as one can 
perceive meteorological subtle variations, which could be of fundamental importance 
for the interpretation of multiagent systems themselves. 

‘Mmm. Looks like rain has stopped. Would be perfect for a walk but it's getting cold 
outside and the wind has turned east. Better stay home!’ 
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Abstract. Most existing 3D virtual worlds are based on a ’’meeting- 
space” model, conceived as an extension of textual chat systems. An at- 
tractive 3D interface is assumed to be sufficient to encourage the emer- 
gence of a community. We look at some alternative models of virtual 
communities, and explore the issue of promoting community formation 
in a 3D meeting space. We identify some essential or desirable features 
needed for community formation - Identity, Expression, Building, Per- 
sistence and Focus of Interest. For each, we discuss how the requirements 
are met in existing text-based and 3D environments, and then show how 
they might be met in future 3D virtual world systems. 



1 Introduction 

he history o the nternet is one o d pt tion ools nd te hnologies on eived 
or serious purposes - ele troni m il senet the orld ide e - h ve een 
swi tly seized upon y ordin ry users nd used to ex h nge person 1 in orm tion 
or in orm tion rel ted to person 1 interests s mu h s rrier o s ientifi 
in orm tion the nternet is h nnel or so i 1 inter tions 

his new underst nding o the nternet s so i 1 medium onstitutes si 
ssumption or m ny developers o rowsers or 3 virtu 1 worlds nvironments 
su h s 1 xxun nter tive’s ommunity ryo’s euxieme onde nLive 
e hnologies’ r veller nd Sony’s ommunity 1 e re 11 sed on simil r 
model more or less re listi visu 1 world in whi h people meet to so i lise 
hile the rowsers Iso lend themselves to other uses the show se ppli tions 
re typi lly sed on this ide o virtu 1 worlds s so i 1 sp es; homes or 
virtu 1 ommunities th t will emerge nd exist in the ontext o the virtu 1 
world 

e intend to explore two m in issues irst we propose si 1 ssifi tion 
o some di erent types o nternet- sed ommunities his will le d us to sk 
some si uestions out the w ys th t 3 virtu 1 world systems re used nd 
whether those uses re lly represent the most ppropri te ones or su h systems 
he se ond se tion o the p per will suggest some re uirements or systems 
designed to support virtu 1 ommunities e will look t the w y these re uire- 
ments re met in existing 3 virtu 1 world systems t e tures o e rlier systems 
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th t might e d pted or use in 3 virtu 1 worlds nd t uni ue e tures o 
3 virtu 1 worlds th t ould e exploited to pit lise on the strengths o the 
medium 

2 Communities on the Internet 

he su ess o the nternet in ostering virtu 1 ommunities in world in whi h 
re 1 ommunities re e oming in re singly r gmented nd unst le h s not 
gone unnoti ed s o Ro kwell o 1 xxun puts it 

” ommunity is eing seen s the long-sought ’killer pp’ th t will 
turn the nternet into profit le resour e ” 

he desire to re te ( nd exploit) ’ ommunity’ ppe rs to underlie m ny o 
tod y’s 3 virtu 1 world systems he ommon h r teristi s o su h systems 

n e riefly st ted s ollows the 3 world is designed s virtu 1 meeting 

pi e reminis ent in some import nt w ys o our own world sers onne t 
inter t nd ommunity emerges he promotion o 3 virtu 1 world systems 
or this use ppe rs to e motiv ted y the ssumption th t the ’ mili rity’ o 

the world with its physi 1 sp es nd em odied v t rs will m ke it more 

essi le nd intim te th n more str t environments 

here re num er o ssumptions here th t seem to re uire urther ex mi- 

n tion; one import nt one is th t we n hope to m ke the virtu 1 world enough 
like the re 1 world th t mili rity tors will work in our vour ( nd th t the 
overhe d o the simul tion won’t work g inst us y m king the tool too slow or 
wkw rd to use) nother is th t inter tion in n r itr ry ontext is y itsel 
suffi ient to re te ommunity something whi h is not ne ess rily the se 

ore su tie is the ssumption th t this is the only or the most ppropri te 
model or ommunity orm tion nd the est model or the ppli tion o 3 
virtu 1 world te hnologies n the rem inder o this se tion we will ex mine 

some Item tive models (shown s hem ti lly in igure ) nd dis uss their 

impli tions or 3 virtu 1 worlds 

2.1 Shared Interest Communities 

ommon type o nternet ommunity is sed on the ex h nge o in orm tion 
out sh red topi o ommon interest Su h ommunity o ten Ire dy exists 

in potenti 1 orm he worldwide popul tion o imi endrix ns or inst n e 
is potenti 1 net ommunity he nternet provides w y or the mem ers 
to ommuni te nd or the ommunity itsel - with its roles onventions nd 
ommuni tion p tterns - to ome into eing 

ommunities o this kind re likely to e multi-mod 1 y whi h we me n 
th t they employ v riety o di erent ommuni tion h nnels voured ’ ore 
modes’ in lude m iling lists senet newsgroups nd topi -spe ifi nternet Re- 
1 y h t ( R ) h nnels s the ommunity e omes est lished mem ers ex- 
ploit other h nnels to extend the tivity o the ommunity he ommunity 
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Meeting-space 

communities 

IRC, MOOs 



Shared interest 
communities 

WWW, Usenet, Mailing-lists 
Topic specific IRC channels 



MULTI-MODAL 




Role-adopting 

communities 

MUDS, Themed MOOs 



SINGLE TOOL 



Fig. 1. ommunity types 



sed round the network g me u ke or ex mple h s its own e sites R 
h nnels nd senet newsgroups em ers o gr phi s so tw re m iling list 
K List show o their work on their e sites nd ex h nge te hni ues using 
( n R e uiv lent) ven users o ommunity- o used h t systems m y 
use Item tive h nnels; ( ulti- ser imension je t- riented) users 

ex h nge priv te em il nd users o sever 1 virtu 1 world systems use e rings 
to provide ess to their worlds nd rel ted m teri Is n sense multi-mod 1 
sh red interest ommunities re perh ps the fin 1 st te o ny nternet ommu- 
nity; ny su ess ul ommunity will ultim tely tend tow rds this type 

n p ssing it is worth sking whether 3 virtu 1 worlds might not h ve role 
to pi y s n ’extr mode’ in multi-mod 1 ommunities 3 worlds re typi lly 
presented s m in h nnel or inter tion in ommunity hey might e u lly 
well t s supplement ry h nnel mu h s e sites re tod y his possi ility 
is t present rel tively little explored or supported 

2.2 Role- Adopting Communities 

se ond kind o ommunity might e termed ’role- dopting’ ommunity 
Su h ommunities re typi lly sed ( t le st initi lly) round single mode 
o inter tion nd strong ’theme’ whi h typi lly t kes the orm o story 

or im gin ry ontext rti ip nts ssume roles ppropri te to the ontext nd 

t ke re to ’ t in h r ter’ his is kind o ’sh red tivity’ ommunity ut 
with the di eren e th t the sh red tivity is em edded in the medium o inter- 

tion sers re not using h nnel (em il) to t Ik out ommon interest 

( imi endrix) whi h exists independently o th t h nnel; they re eng ging 
in n inter tion whose orm is determined y the h nnel n this tegory we 
in lude oth online dventure g mes (i e s) nd non-g ming environments 

(i e themed so i 1 worlds); the h r teristi e tures re the import n e o 
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strongly-defined ontext nd o dopting nd m int ining distin five role or 
h r ter 



2.3 Meeting- Space Communities 

istori lly role- dopting ommunities ti ve o ten evolved tow rds third kind 
o environment in whi h the ontext or theme is 1 rgely stripped w y n su h 
’meeting sp e’ ommunities there re no defined topi s o onvers tion nd 
the environment (i ny) serves prim rily de or tive purpose ( Ithough the 
onventions o the ommunity m y m ke it impolite to ignore the ontext too 
1 t ntly see ) or su h ommunity to emerge nd flourish there must e 
1 rge num er o regul r repe t visitors who eng ge in extended inter tions 
ith no spe ifi sh red interest to dr w in nd ret in p rti ip nts the so i 1 
inter tions themselves need to t s the ’glue’ th t inds the ommunity to- 
gether ommunities o this kind re ommonly seen in s R nd simil r 
h t systems hey re Iso the prim ry model or 3 virtu 1 world systems 
he t th t so m ny 3 virtu 1 world systems ppe r to e iming to 
support this kind o ommunity is interesting n the e o it one o the 
gre test strengths o 3 virtu 1 world system is its ility to re te ri h 

environment yet the ’meeting sp e’ model 1 rgely ignores the environment 
uilding ommunity ’in v uo’ is mu h more h llenging t sk th n re ting 
one in the ontext o sh red tivity or interest oreover e use so i 1 
inter tions re und ment 1 to ommunity orm tion tools used must m ke 
inter tions s fluid nd expressive s possi le he m hinery needed to render 
3 world is omput tion lly expensive resulting in sluggish per orm n e on 11 
ut the most power ul h rdw re while the h t window in whi h the inter tion 

t kes pi e is typi lly shrunk to minimum y ontr st n R or 

lient n run on Imost ny m hine nd there is nothing to t ke w y s reen 
sp e rom the text window in whi h the inter tion o urs 

hese o serv tions r ise the ollowing uestions is the ’meeting pi e’ model 
o ommunity ne ess rily the most ppropri te one or exploiting the potenti 1 
o 3 virtu 1 worlds we dopt this model wh t n the 3 world ontri ute 

in terms o improved inter tion u lity whi h n justi y the extr ost o the 

lient re there other te hni ues we ould Iso use to promote ommunity- 
uilding in our virtu 1 worlds 

n the next se tion we will try to look t w ys in whi h we might m ke 3 
virtu 1 world environments more e e tive in promoting ommunity e propose 
five re uirements th t we see s essenti 1 or desir le or ommunity orm tion 
or e ho these we o er justifi tion nd then ex mine how the re uirement 
is met in existing systems oth 3 nd ’tr dition 1’ e then suggest how uture 
3 systems might meet the re uirement dr wing on oth 3 -spe ifi nd more 
gener 1 te hni ues 
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3 Promoting Community in ’Meeting Spaces’ 

ommunity orm tion is omplex usiness ny o the tors th t le d to 
the orm tion o su ess ul ommunity re int ngi le nd nnot e sily e 

engineered Su h tors in lude the o urren e o orm tive events (see 7 ) nd 

the ommitment or dyn mism o individu 1 mem ers ( 9 spe ks o ’keystone 
pi yers’) hether ommunity flourishes or st gn tes will depend on the type 
o inter tions th t t ke pi e within it 

here re however some tors th t m y e men le to ontrol 2 
presents use ul list o re uirements o using hiefly on the omput tion 1 e - 
tures o the supporting environment ere we present our own list o r ther more 
str t re uirements Ithough there will o ten e degree o overl p 

riefly we elieve th t ommunity orm tion re uires th t the environment 
h s some i not 11 o the ollowing properties users must h ve the me ns to 
re te distin tive identity the me ns to use non-ver 1 orms o expression 
nd the me ns to uild sp es nd o je ts he world itsel should e evolving 

nd persistent nd ide lly there should Iso e ommon tivity or o us o 

interest hese re uirements re summ rized s hem ti lly in igure 2 




Identity Expression Building Persistence Shared Interest 

Fig. 2. ive re uirements or ommunity orm tion 



3.1 Identity 

ne o the first e tures th t we see s ne ess ry or the orm tion o ommunity 
is me h nism or mem ers to re te nd m int in n identity (not ne ess rily 
the s me s their re 1-world identity ut nonetheless distin tive nd re ognis- 
le) rgues th t identity is und ment 1 to est lishing the rel tionships 

ne ess ry to ondu ting usiness nd regul ting group eh viour n in ility 
to present ourselves s individu Is will dire tly imp t our ility to orm so i 1 
rel tions 

dentity is multi- eted ho we re is defined y our rel tions with others 
y wh t we s y nd do nd y the w y we present ourselves nd y the things 
th t we own nd ontrol ur rel tions with others re typi lly defined in 
w ys th t re independent o the environment he environment m y ontri ute 
riti lly however in the ilities it provides or users to present themselves t 
is this issue we will onsider here 
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Identity in Other Channels striking e ture o m ny environments sup- 
porting virtu 1 ommunities re the limited me ns they o er or onveying iden- 
tity n ele troni m il senet or nternet Rel y h t ( R ) p rti ip nts re 
judged Imost ex lusively on wh t they write premium is pi ed on u lities 
su h s wit rti ul y origin lity nd mili rity with sh red onventions e- 
yond their writing users n onvey identity only through their ’h ndle’ ( R ) 
or m il ddress (m il or news not Iw ys under the user’s ontrol) nd use o 
sign tures (m il or news) Surprisingly some o the most oherent nd olour ul 
ommunities seem to emerge in pre isely su h limited environments Restri tions 
m y even en our ge re tivity nd ssertion o identity 

Ri her ilities re o ered y ugmented text- sed environments su h s 
s where users m y provide des riptions o themselves ostensi ly physi 1 
ut o ten used to onvey something out the owner’s tu 1 or desired person- 
lity (see 6 ) Someone who des ri es themselves s ’’Short, fair-haired, slightly 
chubby, wearing jeans and a denim jacket. ” is ommuni ting something r ther 
di erent rom someone who des ri es themselves s ”A nervous-looking three- 
legged centaur wearing two Cecil the Seasick Sea-Serpent puppets on its hands. ” 
r phi 1 environments o er di erent possi ilities n 2 visu 1 environ- 
ments su h s 1 e users n sele t gr phi s to represent themselves mu h s 
users sele t textu 1 des riptions hile pi ture is not ne ess rily worth 

thous nd words the use o gr phi s to onvey h r ter ( nd s we sh 11 see 
1 ter mood) n e uite su tie nd expressive 



Identity in 3D Virtual Worlds sers in 3 virtu 1 worlds re represented 
y V t rs gr phi 1 represent tions o their virtu 1 person hese v t rs re 
typi lly pi ked rom set o ustomis le sto k types n ortun tely the r nge 
o V t rs V il le is o ten very limited nd the ustomis tion possi ilities m y 
e too restri ted to ommuni te nything me ning ul out the owner n m ny 
existing 3 environments the limit tions o the v il le v t rs hinder the 
est lishment o ny re 1 identity 



Improving Identity in 3D Virtual Worlds o try to m ke v t rs more 
’identifi le’ there re two possi le ppro hes ne is to orrow rom text- 
sed virtu 1 worlds y tt hing inspe t le textu 1 des riptions to v t rs 
tt hed des riptions n in lude ddition 1 met -in orm tion; sever 1 3 sys- 
tems support ’hyper rds’ virtu 1 usiness rds whi h n rry n mes (re 1 

or im gin ry) m il ddresses nd e site RLs 

t the other end o the s le re te hni ues whi h m ke use o the uni ue 
properties o visu 1 3 worlds ne possi ility is to m p the user’s re 1-world 

likeness to their v t r s either still or moving im ge (e g 3 ) hile this 
might ppe r to resolve the issue o identity in the most dire t w y possi le it 
is un le r how popul r it is likely to e in pr ti e; the lending o the re 1 nd 
the virtu 1 n sometimes d m ge the illusion nd m ny users m y pre er to 

present n str t im gined identity r ther th n their re 1-li e ppe r n e 
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n Item tive is to Ilow users gre ter reedom in the hoi e o v t rs 
ide lly y letting them design or d pt their own his will re uire some st n- 
d rdiz tion to ensure interoper ility n issue whi h is eing ddressed y the 
nivers 1 v t rs e ort o the R L onsortium nvironments m y need to 
en or e onstr ints on v t r properties in order to void n ’ v t r rms r e’; 
the desire or gre ter sel -expression m y le d to more omplex models with n 
inevit le imp t on downlo d times nd r me r tes 

V t r hoi e m y Iso e su je t to estheti onstr ints 3 orre tly 
predi ts th t there re likely to e so i 1 st tus issues rel ted to v t r hoi e 
while entering strongly-themed world with n in ppropri te v t r riding 
sk te o rd nd we ring wr p round sh des in medi ev 1 stle or ex mple 
is likely to use o en e nd resentment 

Itim tely it seems likely th t we will move tow rds truly person lised 
V t rs ut in the me ntime we should not negle t the potent! 1 o non-visu 1 
h nnels (i e textu 1 des riptions nd hyper rds) whi h h ve the dv nt ge o 
pre ision nd expressiveness t very low ost 



3.2 Expression 

Su ess ul so i 1 inter tions depend on gre t de 1 more th n simply ex h ng- 
ing words n e-to- e onvers tions we use p r linguist! ues su h s tone- 
o -voi e i 1 expressions nd ody 1 ngu ge to onvey signifi nt proportion 
o our mess ge le troni environments whi h deny us the use o these ues 
n o ten impose severe onstr ints on ommuni tion we n in some w y 
restore these ues we m y e le to re te ri her nd more n tur 1 inter tions 
th t will en our ge so i 1 onding 



Expression in Other Channels erh ps the est known orm o extr lin- 
guist! ues in online environments re the smileys used origin lly in ele troni 
m il nd news hese onvention lised sym ols serve Imost the s me un tion 

s tone o voi e with v ri tion in me ning ording to ontext he ’h ppy’ 

smiley : - ) my me n th t the user is pie sed with some st te o irs 

just re erred to th t they’re m king joke or more r rely to indi te gener 1 

st te o h ppiness se to sign 1 jokes is widespre d p rti ul rly in ontexts 

when iling to per eive n utter n e s joke ould use o en e 

ext users h ve other onvention 1 devi es v il le to them su h s use 

o upper se to S sterisks or emph sis et hese onventions me 

out e use they were needed; even ele troni m il o ten seems to e loser to 
onvers tion th n letter-writing nd so su stitutes were ui kly ound or the 
extr -linguist! m rkers we use to m ke onvers tion fluid nd un m iguous 
hese devi es re Iso used in R nd s/ s ut su h syn hronous 
inter tions o er ddition 1 possi ilities or expression ilities in the environ- 
ment su h s the ’emote’ omm nd provide w ys to des ri e h r ter tions 
whi h re ommonly used to onvey mood or ttitude his re hes its most 
omplex in so i 1 s or s where onvers tion m y involve el or te 
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ritu lised use o st nd rd omm nds nd o ten humorous inter tion with o - 
je ts in the world (see 5 ) 

r phi 1 systems su h s he 1 e supplement these devi es urther y 
use o visu 1 in orm tion he im ge used to represent h r ter n e 

swit hed t ny time nd this e ture is sometimes used to onvey mood or 
ttitude sers m y h ve li r ry o di erent im ges to dr w on nd sign 1 
their st te o mind or eelings out nother user y su stituting one or n- 
other ( em le user might present neutr 1 im ge to str ngers or inst n e 
nd more re ognis ly em le one to intim tes) 



Expression in 3D Virtual Worlds 3 virtu 1 worlds whi h typi lly in lude 
text- h t element o ten dr ws on the s me onventions s ove Smileys 
re widely used nd in the sen e o emotes sel -des riptive utter n es re 
sometimes seen his my e in p rt e use m ny 3 users h ve ome rom 
text- sed kground ut it m y Iso e e use the expressive tools provided 
y the 3 environment re too rude or wkw rd to use 

xpli it support or expression in 3 rowsers typi lly t kes the orm o 
push- utton ontrols whi h use the user’s v t r to gesture or dopt 
i 1 expression estures re ’one-o ’ movements while i 1 expressions m y 
persist until su stituted y nother 

xpression support in 3 virtu 1 worlds is pro lem ti estures nd i 1 

expressions re o ten ex gger ted nd do not ne ess rily m p well to di erent 
V t r types (i your v t r is fish how do you onvey surprise or h ppiness ) 
oreover it ppe rs th t users do not e sily mix text nd gr phi s yping 
joke nd then li king the smile utton to show th t you’re not serious does 
not seem to ome n tur lly ore seriously the gesture doesn’t ppe r in the 
written re ord; i someone re ds the inter tion 1 ter or misses the gesture they 
h ve no w y o knowing you were joking n su h ir umst n es most users will 
use smiley inste d 



Improving Expression in 3D Virtual Worlds 3 virtu 1 worlds should 
provide us the me ns to restore some o the ody 1 ngu ge th t we lose in 
tr nsition to the ele troni environment ut in pr ti e it h s not worked th t 
w y V t rs in onvers tion tend to ppro he h other orient themselves 
roughly tow rds e h other ( ut not Iw ys) nd rem in motionless throughout 
the inter tion No use is m de o movement or dist n e nLive e hnologies 
h ve ound th t the ody is Imost entirely irrelev nt nd h ve redu ed their 
V t rs to mere he ds e might sk wh t i nything the 3 virtu 1 world 
ontri utes to expressiveness 

Re 1-time video-m pping o ers one possi ility llowing users to show i 1 
expressions oupled with voi e- r ther th n text- sed ommuni tion this 
would give us the possi ility to inter t more n tur lly ut t this point we 

ould st rt to wonder why we need virtu 1 world t 11 11 we’re interested 

in is spee h nd i 1 expressions why not use video- on eren ing 
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n Item tive might e to use v t r ody 1 ngu ge whi h is under indire t 
r ther th n dire t ontrol o the user se o smiley in text ould prompt the 
V t r to grin moment rily sers ould set p r meters to indi te their gener 1 
st te o mind nd the world would respond y nim ting the v t r ontinously 
in su h w y s to suggest the ppropri te mood oredom pprehension 
nxiety et (see 0 ) t rem ins to e seen how e e tive su h systems will e 
in onveying mood in re 1 inter tions ut i we intend to exploit the e tures 
o the 3 world to the m ximum this looks like use ul dire tion to explore 

3.3 Building 

or ommunity to orm users should h ve some kind o involvement in the 
world th t they o upy n h nnels th t support n expli it ’world’ (3 virtu 1 
worlds S nd s r ther th n R or em il) there seems to e n 

instin tive desire or users to ontri ute to the world in some w y uilding not 
only gives the user person 1 st ke in the world ut it n Iso enri h the world 
or other users ( onsider the se mentioned in 9 o user who introdu ed fish 
nd fishing poles to the ue lo edu tion 1 ) 

uilding is Iso n tivity th t helps to orm identity nd en our ge inter- 
tions ’virtu 1 home’ n reve 1 s mu h out us s n v t r des ription 
(see the des ription o r est’s room in 7 ) re tion o use ul or interest- 
ing o je t gives the re tor st tus ( 2 ) while ex h nge o desir le o je ts or 
te hni 1 in orm tion m y orm the sis o inter tions with other users 

uilding h s other roles je ts th t re uilt serve s t ngi le reminder 
o the history or the onventions o the ommunity he re tion o ver s to 
en psul te running jokes in s is good ex mple o this uilt o je ts m y 
Iso serve s the o us or inter tions nd in p rti ul r or the kind o ritu lised 
ver 1 si psti k h r teristi o s 

Building in Other Channels uilding is not e ture o em il news or R 
Some s nd s however permit users to dd new o je ts to the world 
r nging rom urniture to simple utom t to sp es whi h other users n enter 
nd explore je ts re typi lly p rt o ull o je t-oriented system whi h 
Hows new o je ts to inherit (or override) eh viours rom their p rent o je ts 
Some progr mming skill is re uired to define interesting eh viours ut the 
implement tion 1 ngu ges re gener lly simple enough th t new o je ts n e 
dded to the world without too mu h diffi ulty 

Building in 3D Virtual Worlds he n ture o 3 virtu 1 worlds me ns th t 
uilding is likely to e more omplex th n in textu 1 worlds mong the 3 
virtu 1 world systems th t do How some kind o uilding re ryo’s euxieme 
onde nd ir le o ire’s Iph orld euxi erne onde provides e h user 
with ustomis le ’home’ ut the possi ilities ppe r rel tively onstr ined 

nd the emph sis is on sele tion nd onfigur tion r ther th n uilding n m ny 
w ys this p r llels the issue o v t r hoi e dis ussed ove Iph orld is 
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r ther more interesting; users my re te o je ts sed on pre- uilt omponents 
(Renderw re R o je ts) nd ssign them properties visi ility solidity 
nd eh viours re tions to li ks ollisions et 

Improving Building in 3D Virtual Worlds he power nd flexi ility o 
s rises 1 rgely rom the t th t the system is uilt round n o je t- 
oriented model whi h en our ges ode-sh ring nd re-use simil r model will 
pro ly e ne ess ry in the se o 3 virtu 1 worlds i the world is to e e sy 
to extend nd m int in his m y re uire onsider le h nges to the underlying 
r hite ture o the server nd even to the represent tion 1 ngu ge used or the 
world (see ) 

ppropri te uthoring tools will Iso e re uired t might e desir le to 
integr te the tools with the rowser so s to How the user to work dire tly with 
o je ts in the world r ther th n going through uild-test-edit y le 

llowing the user to uild nd dd r itr ry o je ts using the ull expressive 
power o R L nd v (or simil rly omplex implement tion 1 ngu ge) 

poses 1 rge num er o signifi nt te hni 1 pro lems p rti ul rly where effi- 
ien y nd st ility is on erned n the present st te o te hnology it would e 
11 too e sy or user to uild o je ts th t either onsumed huge u ntities o 

lient nd server resour es or used runtime errors on the lient Nevertheless it 

seems re son le to suggest th t this is the dire tion in whi h 3 virtu 1 worlds 
m y ultim tely move 

3.4 Persistence 

llowing uilding le ds ne ess rily to onsider tion o issues o persisten e 
le rly i users expend e ort in onstru ting o je ts they will expe t the o - 
je ts to e there the next time they onne t je t- uilding implies the need 
or persistent stor ge me h nism to re ord 11 h nges m de in the world 

here n e other spe ts to persisten e s well So i 1 inter tions onsti- 

tute kind o persisten e the so i 1 group ts s ’memory’ or p st events 

individu 1 identities sh red onventions nd so orth his is not something 
th t n e sily e engineered nd is in t dire tly rel ted to the growth o 
ommunity s ommunity grows it e omes persistent 

ersisten e n Iso e interpreted in nother w y r doxi lly persistent 
world is one th t evolves over time world whi h is identi 1 e h time we 
onne t to it 1 ks ny re 1 ’existen e’; it seems to exist ’outside time’ nd this 

rozen u lity n detr t seriously rom the ppe 1 o the world 

Persistence in Other Channels s nd s whi h support uilding 

implement o je t persisten e y regul r kups o the d t se y opying 

o je ts to king stor ge it is possi le to ensure th t r shes or server re oots 

will not destroy re ted o je ts 

s nd s Iso provide kind o support or so i 1 persisten e in the 
sh pe o news systems ssues o the d y nd ontri utions rom mem ers o the 
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ommunity my e re orded in re d le orm th t onstitutes in ert in 
sense kind o re ent history o the ommunity 

Persistence in 3D Virtual Worlds or o vious re sons only those worlds 
th t support m king h nges to the world h ve persistent e tures nd only or 
rel tively limited set o properties s r s we re w re no urrent 3 world 
supports the kind o r-re hing nd perv sive persistent model seen in s 
xpli it support or so i 1 persisten e (i e news systems) is typi lly provided 

extern lly i t 11 in the orm o e p ges (the euxieme onde or ex mple 

h s n extensive e - sed news servi e) 

Improving Persistence in 3D Virtual Worlds ully o je t-oriented world 
model will re uire persisten e me h nisms simil r to those ound in s 

hile the implement tion o su h d t se systems is well-understood it re- 

m ins to e seen how e sily it n e d pted to the needs o 3 worlds 

ssues o so i 1 nd ontextu 1 persisten e re less te hni lly h llenging 

ut m y re uire new ppro h to the provision o 3 worlds n p rti ul r 

the t o world- uilding m y need to e seen not so mu h s one-o en- 
gineering tivity ut s pro ess o ontinuous re tion he world uilders 
need to provide their world with n evolving ontext in order to enh n e the 
user’s per eption o eing in world whi h is onsistent nd enduring his m y 

involve not so mu h dding new e tures to the world s exploiting the visu 1 

represent tion to onvey the ide o p ss ge o time d y nd night se sons 
even de y; in short nything th t will help to give the impression o living 
nd h nging world r ther th n mere model 

3.5 Shared Purpose and Activity 

ur 1 st re uirement returns us to our e rlier dis ussion o di erent ommunity 
types t is in some senses desired r ther th n re uired h r teristi so we 
sh 11 not n lyse it in s mu h depth s the others he ex mple o s nd 

R show th t ommunities n emerge nd thrive without ny sh red tivity 

other th n th t o t Iking to others Nevertheless o serv tion suggests th t 

users re likely to e more toler nt o the limit tions o tool i they h ve 

V lid extern 1 re son or using it we w nt people to use our 3 virtu 1 worlds 
inste d o the simpler swi ter h nnel o R we need to look or ppli tions in 
whi h the use o 3 virtu 1 world provides n dded v lue r ther th n merely 
n en um r n e 

inding suit le ppli tions is wide-open rese r h re ne possi ility 

would e to move w y rom the ’meeting sp e’ model tow rds the ’role- 
dopting’ model nd use the power o the 3 world to re te ompelling 
ontext or inter tions (Sony i tures h ve m de n interesting step in this 
dire tion y using the Sony ommunity 1 e system to re te movie tie-in 
worlds in whi h users pi y roles sed on the film) ther ppli tions might 
in lude virtu 1 improvis tion 1 the tre oil or tive work using visu 1 elements 
or multimedi ’j m sessions’ 
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4 Conclusion 

urrent uses o 3 virtu 1 worlds re typi lly sed on ’meeting sp e’ model 

o ommunity his is not the e siest model to exploit the p ilities o 3 

worlds o en our ge ommunity growth in this ontext we must find w ys to 
promote so i 1 inter tions nd involvement in the world o do so we will need 
to dr w on oth existing me h nisms nd dv n ed 3 -spe ifi te hnologies to 
try to m ke our virtu 1 environment more n tur 1 nd ompelling 

e should Iso keep in mind th t su ess ul ommunities o 11 kinds tend to 

evolve into sh red interest ommunities th t m ke use o v riety o di erent 

ommuni tion h nnels hen designing rowsers or 3 virtu 1 worlds it m y 
e use ul to t ke this into ount nd pi n or their use s p rt o olle tion 
o tools r ther th n uni ue h nnel or inter tion 

here is no olden Rule or ommunity uilding Nevertheless use ul rule 
o thum or implementors my e to review e h new un tion lity proposed 
in terms o the five re uirements we h ve identified in this rti le - identity 
expression uilding persisten e nd sh red interest - nd onsider how it might 
ontri ute to e ho them 
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Abstract. We propose a system for controlling in real time the localisation of 
sound sources. The system, called MidiSpace, is a real time spatializer of Midi 
music. We raise the issue of which interface is the most adapted for using 
MidiSpace. Two interfaces are proposed: a 2D interface for controlling the 
position of sound sources with a global view of the musical setup, and a 
3DWRML interface for moving the listener’s avatar. We report on the design 
of these interfaces and their respective advantages, and conclude on the need 
for a mixed interface for spatialization. 



Active Listening 

We believe that listening environments of the future can be greatly enhanced by 
integrating relevant models of musical perception into musical listening devices, 
provided we can develop appropriate software technology to exploit them. This is the 
basis of the research conducted on “Active listening” at Sony Computer Science 
Laboratory, Paris. Active Listening refers to the idea that listeners can be given some 
degree of control on the music they listen to, that give the possibility of proposing 
different musical perceptions on a piece of music, by opposition to traditional 
listening, in which the musical media is played passively by some neutral device. The 
objective is both to increase the musical comfort of listeners, and, when possible, to 
provide listeners with smoother paths to new music (music they do not know, or do 
not like). These control parameters create implicitly control spaces in which musical 
pieces can be listened to in various ways. Active listening is thus related to the notion 
of Open Form in composition [8] but differs by two aspects: 1) we seek to create 
listening environments for existing music repertoires, rather than creating 
environments for composition or free musical exploration (such as PatchWork [11], 
OpenMusic [2], or CommonMusic [18]), and 2) we aim at creating environments in 
which the variations always preserve the original semantics of the music, at least 
when this semantics can be defined precisely. 

The first parameter which comes to mind when thinking about user control on 
music is the spatialization of sound sources. In this paper we study the implications of 
giving users the possibility to change dynamically the mixing of sound sources. In te 
next section, we review previous approaches in computer-controlled sound 
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spatialization, and then propose a basic environment for controlling music 
spatialization, called MidiSpace. We then describe a simple 2D interface for 
controlling sound sources, and then describe a VRML interface which gives users a 
more realistic view on the music heard. We compare the two approaches and argue in 
favor of a mixed solution integrating both interfaces. The last section describes the 
overall design and implementation of the resulting system. 



Music Spatialization 

Music spatialization has long been an intensive object of study in computer music 
research. Most of the work so far has concentrated in building software systems that 
simulate acoustic environments for existing sound signals. These works are based on 
results in psychoacoustics that allow to model the perception of sound sources by the 
human hear using a limited number of perceptive parameters [4]. These models have 
led to techniques allowing to recreate impression of sound localization using a limited 
number of loudspeakers. These techniques typically exploit differences of amplitude 
in sound channels, delays between sound channels to account for interaural distances, 
and sound filtering techniques such as reverberation to recreate impressions of 
distance and of spatial volume. 

For instance. The Spatialisateur IRCAM [10] is a virtual acoustic processor that 
allows to define the sound scene as a set of perceptive factors such as azimuth, 
elevation and orientation angles of sound sources relatively to the listener. This 
processor can adapt itself to any sound reproduction configuration, such as 
headphones, pairs of loudspeakers, or collections of loudspeaker. Other commercial 
systems with similar features have recently been introduced on the market, such as 
Roland RSS, the Spatializer (Spatializer Audio Labs) which allows to produce a 
stereo 3D signal from an 8-track input signal controlled by joysticks, or Q-Sound 
labs’s Q-Sound, which builds extended stereophonic image using similar techniques. 
This tendency to propose integrated technology to produce 3D sound is further 
reflected, for instance, by Microsoft’s DirectX API now integrating 3D audio. 

These sound spatialization techniques and systems are mostly used for building 
various virtual reality environments, such as the Cave [5] or CyberStage [6], [8]. 
Recently, sound spatialization has also been included in limited ways in various 3D 
environments such as Community Place's implementation of VRML [12], ET-h- [1], 
or proprietary, low-cost infrastructures [3]. 

Based on these works, we are interested in exploiting spatialization capabilities for 
building richer listening environments. In this paper, we concentrate on the interface 
issue, i.e. how to give average listeners the possibility of exploiting sound source 
spatialization in a natural, easy way. We will first describe our basic spatialization 
system MidiSpace, which precisely allows user to control in real time the 
spatialization of sound sources. Then we describe two interfaces for MidiSpace, and 
compare their respective advantages. 
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The Basic MidiSpace System 

MidiSpace is a system that gives listeners control on music spatialization. We first 
outline the characteristics of midi-based spatialization before describing the system. 



Midi-Based Spatialization 

MidiSpace is a real time player of Midi files which allows users to control in real time 
the localization of sound sources through a graphical interface (extensions to audio 
are not discussed in this paper). MidiSpace takes as input arbitrary Midi files [9]. The 
basic idea in MidiSpace is to represent graphically sound sources in a virtual space, as 
well as an avatar that represents the listener itself. Through an appropriate editor, the 
user may either move its avatar around, or move the instruments themselves. The 
relative position of sound sources and the listener’s avatar determine the overall 
mixing of the music, according to simple geometrical rules illustrated in Fig. 1. The 
real time mixing of sound sources is realized by sending Midi volume and panoramic 
messages. 



sound source 




listener’s 

avatar 



Fig. 1. Volume of sound_sourcej = f(distance(graphical-object, listener_avatar)). f is a function 
mapping distance to Midi volume (from 0 to 127). Stereo position of sound source i = 
g(angle(graphical_Objectj, listener_avatar)), where angle is computed relatively to the vertical 
segment crossing the listener’s avatar, and g is a function mapping angles to Midi panoramic 
positions (0 to 127). 

It is important to understand here the role of Midi in this project. On the one hand, 
there are strong limitations of using Midi for spatialization per se. In particular, using 
Midi panoramic and volume control changes messages for spatializing sounds does 
not allow to reach the same level of realism than when using other techniques (delays 
between channels, digital signal processing techniques, etc.), since we exploit only 
difference in amplitude in sound channels to create spatialization effects. However, 
this limitation is not important in our context for two reasons : 1) this Midi-based 
technique still allows to achieve a reasonable impression of sound spatialization 
which is enough to validate our ideas in user interface and control, and 2) more 
sophisticated techniques for spatialization can be added in MidiSpace, independently 
of its architecture. 
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We will now describe the interfaces for MidiSpace: first, a 2D interface, which 
provides a global view on the musical setting, and allows to move sound sources 
around. Then we describe a VRJVIL interface and discuss its relevance for music 
listening. We conclude on the interest of a mixed approach. 



The 2D Interface of MidiSpace 

In the 2D interface of MidiSpace, each sound source is represented by a graphical 
object, as well as the listener’s avatar (see Fig. 2. ). The user may basically play a 
midi file (with usual tape recorder-like controls), and move objects around. When an 
object is moved, the spatializer is called with the new position of the object, and the 
mixing of sound sources is recomputed accordingly. Other features allow to mute 
sound sources, or select them as “solo”. 




Listener’ s avatar 



Fig. 2. The 2D Interface of MidiSpace. Here, a tune by Bill Evans (Jazz trio) is being 
performed 

In an initial version, we allowed both sound sources and the avatar to be moved. 
However this was confusing for users. Indeed, moving sound sources amounts to 
changing the intrinsic mixing of the piece. For instance, moving the piano away will 
changes the relationship between the piano sound and the rhythmic part. Moving the 
avatar simply amounts to changing the mixing in global way, but respects the 
relationships between the sound sources. The effect is quite different since in the 
second case the structure of the music is modified. 

The interface provides a global view on the musical setup, which is very 
convenient to edit the musical setting. However, there is no impression of musical 
immersion in the musical piece ; the interface is basically a means for editing the 
piece, not to explore it. 
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The VRML Interface for Navigating in MidiSpace 

Second, we have experimented with interfaces for navigating in the musical space. 
Several works addressed the issue of navigating in virtual worlds with spatialized 
sounds. The most spectacular are probably the Cave system [5] or CyberStage [8], in 
which the user is immersed in a fully-equipped room with surrounding images and 
sound. Although the resulting systems are usually very realistic, their cost and 
availability are still prohibitive. 

Instead, we chose to experiment with affordable, wide-spread technology. A 3D 
version of MidiSpace in VRML has been built (see Fig. 3. ), in which the VRML 
code is automatically generated from the 2D interface and the Midi parser. The idea is 
to create a VRML world in which the user may freely navigate using the standard 
VRML commands, while listening to the musical piece. Each instrument is 
represented by a VRML object, and the spatialization is computed from the user 
current viewpoint. In this interface, the only thing the user can do is move around 
using standard commands; sound sources cannot be moved. 





Fig. 3. MidiSpaceA^RML on the Jazz trio, corresponding to the 2D Interface of Fig. 2. On the 
left, before entering, on the right, inside the club. 



Although the result is more exciting and stimulating for users than the 2D 
interface, it is not yet fully satisfying because the interface gives too little information 
on the overall configuration of instruments, which is a crucial parameter for 
spatialization. When the user gets close to an instrument, she loses the sense of her 
position in the musical set up (e.g. the jazz club, see Fig. 3. ). Of course, this problem 
is a general problem with VRML interfaces, and is not specific to MidiSpace, but in 
the context of a listening environment, it is particularly important to provide a 
visualisation which is consistent with the music being played. This consistency is 
difficult to achieve with a 3D virtual interface on a simple screen. 
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The Mixed Interface 

Based on experiments with users on the two interfaces, we concluded on the interest 
of combining them for obtaining an optimal satisfaction on user control. The 2D 
interface is used for editing purposes, i.e. moving sound sources. Moving the avatar is 
not possible in this interface. The VRML interface is used for exploration, in a 
passive mode, to visualize the musical setting in 3D, and move the avatar around. 
Moving objects is not possible. 

The communication between the two interfaces is realized through the VRML/Java 
communication scheme, and ensures that when the avatar is moved in the VRML 
interface, the graphical object of the 2D interface is moved accordingly. The overall 
architecture of MidiSpace is illustrated in Fig. 4. 




Fig. 4. The architecture of MidiSpace User Interaction. 



Implementation 

The implementation of the MidiSpace Spatializer consists in 1) translating Midi 
information into a set of objects within a temporal framework, 2) scheduling these 
temporal objects using a real time scheduler, 3) the interfaces. 



The Parser 

The Parser task is to transform the information contained in the Midi file into a 
unified temporal structure. The temporal framework we use is described in [18], an 
object-oriented, interval-based representation of temporal objects. In this framework. 
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each temporal object is represented by a class, which inherits the basic functionalities 
from a root superclass TempomlObject. 

One main issue the Parser must address comes from the way Midi files are 
organized according to the General Midi specifications. Mixing is realized by sending 
volume and panoramic Midi messages. These messages are global for a given Midi 
channel. One must therefore ensure that each instrument appears on a distinct Midi 
channel. In practice, this is not always the case, since Midi tracks can contain events 
on different channels. The first task is to sort the events and create logical melodies 
for each instrument. This is realized by analysing program change messages, which 
assign Midi channels to instruments, thereby segmenting the musical structure. The 
second task is to create higher level musical structures from the basic musical objects 
(i.e. notes). The Midi information is organized into notes, grouped in melodies. Each 
melody contains only the notes for a single instrument. The total piece is represented 
as a collection of melodies. A dispatch algorithm ensures that, at a given time, only 
one instrument is playing on a given Midi channel. 



Scheduling Temporal Objects 

The scheduling of MidiSpace objects uses MidiShare [14], a real time Midi operating 
system, with a Java API. MidiShare provides the basic functionality to schedule 
asynchronously, in real time, Midi events, from Java programs, with 1 millisecond 
accuracy. More details on the scheduling are given in [7]. 



MidiSpace Interfaces 

The 2D interface uses the standard Java awt library, and follows a straightforward 
object-oriented interface design. The VRML interface is generated automatically 
from the Parser. More precisely, the Parser generates a file, which contains basically 
1) the information on the global setup, 2) description of individual sound sources, 
corresponding to the various sound tracks identified, and 3) routing expressions to a 
spatializer object, which is defined as a Java script, using the VRML/Java 
communication scheme [12]. An excerpt of the VRML code is shown in Fig. 5. 



Worldinfo {title "Trio jazz"} 

# Various global settings 

Navigationinf o (speed 2 type [ "WALK" ] } 

# The viewpoint from which the spatialization is computed 
DBF USERVIEWPOINT Viewpoint (position -13 0 45} 

# The definition of the musical setting (here, a Jazz 
Club) 
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# the Label 
Transform { 

translation -2 4.7 21 
children [ 

Shape { 

appearance Appearance { 
material Material { 
diffuseColor 111}} 
geometry Text { 

string "Jazz Club"}}]} 



# The sound sources, as identified by the Parser in 
General Midi 
DEF PIANO Transform { 
children [ 

Shape { 

appearance Appearance { 
texture ImageTexture { 
url "piano.jpg" 

repeats FALSE 
repeatT FALSE} 

textureTransf orm TextureTransform {} 

} 

geometry Box { 
size 3 3 3} } ] } 

DEF DRUMS Transform { ...} 

DEF BASS Transform {...} 



# The Java script for handling movements and 
spatialization 
DEF MY_SCRIPT Script { 
url "MusicScript . class " 

field SFString midiFileName 

"http : / /intweb . csl . Sony . f r/~demo/ trio . mid" 
field SFNode channel 10 USE DRUMS 
field SFNode channel2 USE BASS 
field SFNode channel! USE PIANO 
eventin SFVec3f posChanged 
eventin SFRotation orientation 
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eventOut SFRotation keepRight 
eventOut SFVec3f keepPosition} 

" The routing of VRML messages to the Java script 



ROUTE DETECTOR . position_changed TO 

MY_SCRIPT.posChanged 

ROUTE MY_SCRIPT . keepPosition TO 

USERVIEWPOINT. set_position 

ROUTE DETECTOR . orientation_changed TO 

MY_SCRIPT . orientation 

ROUTE MY_SCRIPT. keepRight TO 

USERVIEWPOINT . set_orientation 



Fig. 5. The generated VRML code for describing the musical setting from a given 
midi file 



Conclusion, Future Works 

The MidiSpace system shows that it is possible to give some degree of freedom to 
users in sound spatialization, through an intuitive graphical interface. We argue that a 
unique interface is not appropriate for both moving sound sources and avatars, while 
giving users a realistic feeling of immersion. In the case of spatialization, although 
they appear at first to be similar user actions, moving sound sources and moving the 
avatar bear significantly different semantics, and we concluded that allowing these 
two operations in the same interface is confusing. We propose an approach 
combining a standard 2D interface, appropriate for moving sound sources and editing 
the setup, and a VRML interface appropriate for moving avatars and exploring the 
musical piece. The prototype built so far validates our approach, and more systematic 
testing with users is in progress. 

Future work remains to be done in three main directions. First we are currently 
adding a constraint-based mechanism for maintaining consistency in sound source 
positions [15]. This mechanism allows users to navigate in a restricted space, which 
will always ensure that some mixing properties of the music are satisfied. Second, an 
audio version of MidiSpace is in progress, to 1) enlarge the repertoire of available 
music material to virtually all recorded music, and 2) improve the quality of the 
spatialization, using more advanced techniques such as the ones sketched in the 
introduction of this paper. Finally, an annotation format is currently being designed 
to represent information on the content of musical piece, to enrich the interaction. 
These annotations typically represent information on the structure of the piece, its 
harmonic characteristics, and other kinds of temporal information [17]. These 
extensions are in progress, and remain compatible with our choice for user interface 
design proposed here. 
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Abstract. The Australian Great Barrier Reef is a natural wonder of our world 
and a registered UNESCO World Heritage site hosting 1.5 million visitor-days 
in 1994/95. Tourism is currently the main commercial use and is estimated to 
generate over $1 billion annually.[l] With the coming 2000 Olympics in 
Australia, tourism increases will substantially present a major conservation and 
preservation problem to the reef. This paper proposes a solution to this problem 
through establishing a virtual reality installation that is interactive and evolving, 
enabling many visitors to discover the reef through high quality immersive 
entertainment. This paper considers the technical implications required for a 
system based in Complexity: a distributed DOME and CAVE architectural 
system; a mixed reality environment; artificial life; multi-user interactivity; and 
hardware interfaces. 



1 Introduction 

"Well, life emerged in the oceans, " he adds, "so there you are at the edge [diving off 
the continental shelf], alive and appreciating that enormous fluid nursery. And that's 
why the 'edge of chaos' carries for me a very similar feeling: because I believe life 
also originated at the edge of chaos. So here we are at the edge, alive and 
appreciating the fact that physics itself should yield up such a nursery... " 

- Chris Langton, artificial life pioneer, commenting on Complexity.^ 



' Waldrop, M. Mitchell, 1992 Complexity, The Emerging Science at the Edge of Order and 
Chaos, page 231. Simon & Schuster, 1992. 

J.-C. Heudin (Ed.): Virtual Worlds 98, LNAI 1434, pp. 323-336, 1998 
© Springer- Verlag Berlin Heidelberg 1998 
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1.1 A Cinematic Narrative Virtual Walkthrough 



Jack, Jill and Tanaka have entered the Virtual Barrier Reef Exhibition, where 
"Nintendo" meets classroom. Before going into the environment, they have to choose 
from several different experiential options: Jack picks the "Marine Biologist," Jill 
picks the "Dolphin" and Tanaka the "Snorkeler." 




Fig. 1. Overview of the Virtual Great Barrier Reef installation consisting of a 15 meter 
diameter DOME, Interactive WALLs, 3 CAVE systems and a motion simulator. 



When they put on their equipment mainly consisting of 3D shutter glasses, audio 
communication and other interactive devices, they enter the Dive Vehicle (motion 
simulator) that takes them on a small "journey" to the reef. The Immersion hatch 
opens and they exit into the main DOME [Fig. 1] finding themselves immersed in 20 
meters (60 feet) of water. The main DOME [2] (15 m diameter x 15m high, 360° x 
1 80°), enables them to see fish and other species swim about in 3D. They see a mix of 
autonomous fish, avatars and avatar guides that assist, guide and role-play. They hear 
soundscapes of the reef. They can also see 3 CAVEs [3]. They were told that the 
installation was directly linked to the real reef to affect the climatic conditions within 
the installation. The installation's environment was persistent, evolving and adapting 
just like the real reef. 

Jack finds a school of fish that holds his interest, and points to it using the "laser 
and" pointer on top of his Dive Panel. The panel displays the information about the 
fish, life habits, predators, eating, mating habits, etc. Then, a big grouper swims to 
him (guide avatar) and engages him in a conversation about the reef. A dolphin joins 
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in and he finds out its Jill, and they are able to communicate together through the 
group communication channel with their headsets. 

Tanaka finds an autonomous fish guide and several descriptor icons floating 
nearby. He selects the Japanese version. He inserts his smart card in the nearby stand, 
then experiments with the inter-related lives of the species inhabiting that community 
by moving the various avatars around with a Fish data Glove. At one point a mythical 
aboriginal character swims by and heads for one of the CAVEs. At the warning beep 
on each of their timers, Jill and Tanaka to return to the visitor's center. Jack extends 
his visit by buying more air directly through his smart card and Dive Panel. Once in 
the visitors' center, Jill and Tanaka insert their smart cards to get a color print out of 
their activities during their visit. 



1.2 The Complex Architecture 

The system is a large distributed network based upon proven, stable DOME and 
CAVE technologies. Similar reference models in which this project is based upon 
can be found in the past projects of NICE [4] and ROBOTIX: MARS MISSION [5]. 

The NICE project is a collaborative CAVE-based environment where 6 to 8 year 
old children, represented as avatars, collaboratively tend a persistent virtual garden. 
This highly graphical, immersive virtual space has primarily been designed for use in 
the CAVE, a multi-person, room-sized virtual reality system. As the CAVE supports 
multiple simultaneous physical users, a number of children can participate in the 
learning activities at the same time. 

Interactive DOME projects including the Carnegie Science Center's "ROBOTIX: 
MARS MISSION, was the first example of Interactive Virtual Reality Cinema. Each 
audience member had a three button mouse device that allowed them to make group 
decisions on story and piloting spacecraft. The graphics were rendered in real-time, 
called from an extensive database of Mars's terrain from NASA. The partial dome sat 
35 people, and was 200 degrees wide and 60 degrees high. The system used a SGI 
ONYX, with 4 RIOOOO processors, Infinite Reality Engine, Audio Serial Option 
board, and display boards. The polling system, custom built, fed into a PC which 
talked to the SGI. Majority vote ruled. The dome display used 3 state of the art 
Electrohome RGB projectors. Most important, only one graphic pipeline was used, 
and split into three views using custom built software and edge blending. The overlap 
on edges was 25%, and it appeared seamless and no distortion. Performance, 30-90-n 
fps, no lag. Audio was 3-D spatialized, custom built software, with floor shakers. It 
ran for 9 months, no glitches. Effective, there were some people who thought they 
went to Mars. 



1.3 Methodology of Application 

Whilst the true objective of this installation is to install a strong sense of conservation 
and preservation to the future visitors of the Great Barrier Reef, it seeks to apply it 
through a Constructionist [6] and "Immersive role-play" educative model steeped 
heavy in "Nintendo" style interaction of total consciousness submersion. Thus, our 
methodology objectives are simple - give the visitor an experience that they 
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constructed for themselves, immerse them in wonder and inspiration, give them a 
reason to care, in their minds really take them there, and finally, afterwards give them 
something material they can use to prove and remember their experience. 



2 Distributed DOME and CAVE Architectural System 



2.1 The DOME Zone 

The DOME zone is constructed of CG and real-time/pre-captured video which is 
projected onto the scrim of the DOME together in layers to assist with overall 
processing quality of CG and simulations. The background is a mix of live or pre- 
recorded scenes in the reef, and the CG fish are projected bright enough over the top 
of it to create the mixed illusion. Of course there will have to be certain areas, such as 
coral and other big blocks of CG that have to be animated, but these aren't that 
computationally intensive as they update on a slightly slower level. 




CAVE 

|[xoo« 



DOME 
[zone 1] 



(zen»7) 



virtua Barrier Reef 



Server/Router 



CAVE 

Router 



Webserv 

Router 



Fig. 2. Distributed Relationship of the DOME, CAVES, WALLs and the Internet. 



2.2 Multiplexing and Mixed Topologies 

Since the DOME is the central "Hub" [Eig. 2] of the environment physically, and 
holds the most connectivity to all other zones, we will use it as the focus to consider 
the many facets of multiplexing within the environment and the various other network 
topologies and protocols that connect to it. 
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2.3 The WALL Zones (4 WALLs) 

The walls will be the most computational expensive of all the zones in the 
environment as most of the direct interactivity will happen around them. Walls are 
interactive in that they might have species that will react when visitors walk by them. 
Example would be an alife moray eel might come out to feed, and when a visitor 
walks up to it quickly, a sensing device gives it spatial reaction information. Spot 
projectors and smaller independent computer systems are displaying various small 
interactive displays, while the meta environment was separate and being processed by 
the SGI system. 



2.4 CAVE Zones: (3 CAVEs) 

CAVEs are separate zones or environments that are story related. They can host 
indigenous or mythical stories related to the reef and its inhabitants. Here the visitors 
can have a more immersive, intimate experience with the denizens of the reef, as the 
sea creatures swim and interact with the visitors within the 3 meter cube of each 
CAVE. 

The CAVEs use the standard network topologies and protocols being developed as 
part of the CAVERN network [7] to deal with the multiple information flows 
maintaining the virtual environments. 

Each of the CAVEs, WALLs, and the DOME making up the environment will be 
connected via a high-speed network, allowing information such as avatar data, virtual 
world state / meta data and persistent evolutional data to pass between them. This will 
allow avatars in the CAVEs to swim over onto the WALLS or the DOME, and allow 
museum guides in the guise of reef inhabitants to interact with the visitors. This will 
also allow remotely located CAVE users in the United States, Europe, and Asia to 
join into the collaboration. Similar connectivity has been used in the NICE 
environment [8] to link multiple CAVE sites in the United States to Europe and 
Japan. 



3 Mixed Reality Environment 

The environment of this project uses video, lighting, and sound techniques to 
establish an environment that convinces the user that they are indeed immersed in the 
Great Barrier Reef. 

Video: Video will be used in strategic places includes some parts of the walls and 
the floor. The main imaging of the floor is video, with spot CG projections to provide 
a better illusion of immersion and non-repetition. 

Sound: immersive and 3D in its localization, the sound system provides authentic 
sounds from autonomous life and "extra spatial" depth sounds. 
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3.1 Real-Time Connectivity to the Reef 

Climatic Interference: The virtual model is connected real-time to a section of the real 
reef so that it can automatically adjust the virtual model. Temperature, current, air 
pressnre and other climatic conditions on the reef will cause several actions in the 
virtual environment, such as temperature of the room, flow of the virtual debris, 
virtual water currents, etc. 



3.2 Complex and Evolving Environment 



This environment is considered to be a "living entity", possessing an authentic as 
possible simulation to the real reef. 

For simnlating simple fluid flow, we consider techniques similar to the ones used 
by Wejchert and Haumann Wejchert 91 [9] [Fig. 3] for animating aerodynamics. 
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Fig. 3. The flow primitives (reproduced from the paper by Wejchert and Haumann) 

Assuming inviscid and irrotational fluid, we can construct a model of a non- 
turbulent flow field with low computational cost using a set of flow primitives. These 
include uniform flow where the fluid velocity follow straight lines; source flow— a 
point from which fluid moves out from all directions; sink flow which is opposite to 
the source flow; and vortex flow where fluid moves around in concentric circles. This 
process is also directly linked with the real reef current flow that is constantly 
supplying real-time information. Both of the processes integrated together enable 
lower computation reqnirements and increase the authenticity of the virtual 
environment. 

Much of the seaweed, plankton, coral and other movable plant/marine life are 
simulated and respond dynamically to water currents and movements of other life in a 
realistic manner. In this case it is proposed to use a system which builds the plants in 
a mass-spring chain assembly. In order to obtain computational efficiency, we do not 
calculate the forces acting on the geometric surface of each plant leaf, rather, we 
approximate the hydrodynamic force at each mass point. 



4 Artificial Life 

The artificial life treatment in this project is an exciting and innovative one. As 
mentioned earlier, one of the educative goals in this project is to enable people to 
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"think like a fish" so they can experience first hand through the "eyes of a fish" what 
it's like to live on the reef. 

To achieve this, we have developed two types of models for Alife. The first model 
is a totally autonomous animal that behaves according to the events in the 
environment. It has a life span, is integrated into the ecological system, and possesses 
authentic behaviors of its real life counterpart. 

The second model is a similar model, only that the "brain and motor skills" are 
detached and placed into the control of a visitor using a dataglove to direct the fish. 
The innovation here is that even though a visitor is controlling the "will and motor" of 
the artificial fish, it still has behavior and characteristic traits that will not allow you 
to act outside of the character set for that fish. 



4.1 Artificial Animals for Life-Like Interactive Virtual Reef Creatures 

What's the key to bringing you a captivating experience as a tourist of the virtual 
Great Barrier Reef? Whether you are merely observing the schools of sardines, 
playful dolphins and vicious sharks, or you are living the aquatic life by playing the 
role of one of the virtual reef dwellers (want to be a dolphin, parrot fish or maybe a 
reef shark? Sure you can), realism is the password to the ultimate immersive 
experience. Here we are not just talking about visual authenticity in the appearance of 
the digital sea creatures, but also in their physical environment, and most importantly, 
in the way they move, perceive and behave. In an interactive environment like ours, 
behavioral realism, especially, autonomy of the virtual creatures is indispensable. 

How are we to achieve such integrated realism? Our answer is to build Artificial 
Life (or Alife) models of the aquatic animals. The properties and internal control 
mechanisms of these artificial animals should be qualitatively similar to those of their 
natural counterparts. Specifically, we construct animal-like autonomy into traditional 
graphical models by integrating control structures for locomotion, perception and 
action. As a result, the artificial fishes and mammals have 'eyes' and other sensors to 
actively perceive their dynamic environments; they have 'brains' to interpret their 
perception and govern their actions. Just like real animals, they autonomously make 
decisions about what to do and how to do in everyday aquatic life, be it dramatic or 
mundane. This approach has been successfully demonstrated by the lifelike virtual 
undersea world developed in [10, 11]. 

We tackle the complexity of the artificial life model by decomposing it into three 
sub-models: 

• A graphical display model that uses geometry and texture to capture the form 
and appearance of any specific species of sea animal. 

• A bio-mechanical model that captures the physical and anatomical structure 
of the animal’s body, including its muscle actuators, and simulates its 
deformation and dynamics. 

• A brain model that is responsible for motor, perception and behavior control 
of the animal. 

As an example of such an artificial life model, [Fig. 4] shows a functional overview 
of the artificial fish that was implemented in [10, 11]. As the figure illustrates, the 
body of the fish harbors its brain. The brain itself consists of three control centers: the 
motor center, the perception center, and the behavior center. These centers are part of 
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the motor, perception, and behavior control systems of the artificial fish. The function 
of each of these systems will be previewed next. 




ilrirtn /Jtiind 



Fig. 4. Diagram of an autonomous and avatar model fish 



4.2 Motor System 

The motor system comprises the dynamic model of the sea creature, the actuators, 
and a set of motor controllers (MCs) which constitutes the motor control center in the 
artificial animal's brain. Since our goal is to animate an animal realistically and at 
reasonable computational cost, we seek to design a mechanical model that represents 
a good compromise between anatomical consistency, hence realism, and 
computational efficiency. The dynamic fish model [12, 13] represents a good 
example. It is important to realize that adequate model fidelity allows us to build 
motor controllers by gleaning information from bio-mechanical literature of the 
animal [14, 15] Motor controllers are parameterized procedures, each of which is 
dedicated to carrying out a specific motor function, such as "swim forward", "turn 
left" or "ascend". They translate natural control parameters such as the forward speed, 
angle of the turn or angle of ascent into detailed muscle or fin/leg actions. Abstracting 
locomotion control into parameterized procedures enables behavior control to operate 
on the level of motor skills, rather than that of tedious individual muscle/limb 
movements. The repertoire of motor skills forms the foundation of the artificial 
animal's functionality. 
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Given that our application is in interactive VR systems, locomotion control and 
simulation must be executed at interactive speed. Physics-based dynamic models of 
animals offer physical realism but require numerical integration. This can be 
expensive when the models are complex. 

There are various ways to speed things up. One way is to have multiple models of 
increasing complexity for each species of animal. For example, the simplest model 
will just be a particle with mass, velocity and acceleration (which still exhibits basic 
physical properties and hence can automatically react to external forces, such as water 
current, in a realistic manner). The speed-up can then be achieved by way of 'motion 
culling’, where when the animal is not visible or is far away, only the simplest model 
is used and the full mechanical model is used only when the animal is near by. This 
method, however, is view-dependent and can be tricky when the number of users is 
large. Another way is to use bio-mechanical models whenever possible (when they 
are simple enough to run the simulation at real time) and with more complex animals, 
we can build pre-processed, parameterized motion libraries (which can be played 
back in real time), like what is used by most of today's games. These (canned) motion 
libraries can be built from off-line simulations of elaborate bio-mechanical models of 
the animal or from motion-captured data. One thing to keep in mind is that, no matter 
what the underlying locomotion model is, its interface to the behavior system should 
be kept comparable as parameterized procedures of motor skills. 



4.3 Perception System 

Perception modeling is concerned with: 

1. Simulating the physical and optical abilities and limitations of the animal's 
perception. 

2. Interpreting sensory data by simulating the results of perceptual information 
processing within the brain of the animal. 

When modeling perception for the purposes of interactive entertainment, our first 
task is to model the perceptual capabilities of the animal. Many animals employ eyes 
as their primary sense organ and perceptual information is extracted from retinal 
images. In a VR system, such "retinal" images correspond to the 2D projection of the 
3D virtual world rendered from the point of view of the artificial animal's "eyes". 
However, many animals do not rely on vision as their primary perceptual mode, in 
which case vision models alone may not be able to appropriately capture the animal's 
perceptual abilities. 

It is equally important to model the limitations of natural perception. Animal 
sensory organs cannot provide unlimited information about their habitats. Most 
animals cannot detect objects that are beyond a certain distance away and they usually 
can detect moving objects much better than static objects [14]. If these properties are 
not adequately modeled, unrealistic behaviors may result. 

Moreover, at any moment in time, an animal receives a relatively large amount of 
sensory information to which its brain cannot attend all at once. Hence there must be 
some mechanism for deciding what particular information to attend to at any 
particular time. This process is often referred to as attention. The focus of attention is 
determined based upon the animal's behavioral needs and is a crucial part of 
perception that directly connects perception to behavior. 
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Unfortunately, it is not at all well understood how to model animal sensory organs, 
let alone the information processing in the brain that mediate an animal's perception 
of its world. Fortunately, for our purposes, an artificial animal in its virtual world can 
readily glean whatever sensory information is necessary to support life-like behavior 
by directly interrogating the world model and/or exploiting the graphics-rendering 
pipeline. In this way, our perception model synthesizes the results of perception in as 
simple, direct and efficient a manner as possible. 

The perception system relies on a set of on-board virtual sensors to provide sensory 
information about the dynamic environment, including eyes that can produce time- 
varying retinal images of the environment. The brain's perception control center 
includes a perceptual attention mechanism which allows the artificial animal to train 
its sensors at the world in a task-specific way, hence filtering out sensory information 
superfluous to its current behavioral needs. For example, the artificial animal attends 
to sensory information about nearby food sources when foraging. 



4.4 Behavior System 

The behavior system of the artificial animal mediates between its perception 
system and its motor system. An intention generator, the animal's cognitive faculty, 
harnesses the dynamics of the perception-action cycle and controls action selection in 
the artificial animal. The animator establishes the innate character of the animal 
through a set of habit parameters that determine whether or not it likes darkness or 
whether it is a male or female, etc. Unlike the static habits of the animal, its mental 
state is dynamic and is modeled in the behavior system by several mental state 
variables. Each mental state variable represents a distinct desire. For example, the 
desire to drink or the desire to eat. In order to model an artificial animal's mental state, 
it is important to make certain that the modeled desires resemble the three 
fundamental properties of natural desires: (a) they should be time varying; (b) they 
should depend on either internal urge or external stimuli or both; (c) they should be 
satisfiable. The intention generator combines the habits and mental state with the 
incoming stream of sensory information to generate dynamic goals for the animal, 
such as to chase and feed on prey. It ensures that goals have some persistence by 
exploiting a single-item memory. The intention generator also controls the perceptual 
attention mechanism to filter out sensory information unnecessary to accomplishing 
the goal in hand. For example, if the intention is to eat food, then the artificial animal 
attends to sensory information related to nearby food sources. Moreover, at any given 
moment in time, there is only one intention or one active behavior in the artificial 
animal’s behavior system. This hypothesis is commonly made by ethologists when 
analyzing the behavior of fishes, birds and four-legged animals of or below 
intermediate complexity (e.g. dogs, cats) [15, 16]. At every simulation time step, the 
intention generator activates behavior routines that input the filtered sensory 
information and compute the appropriate motor control parameters to carry the animal 
one step closer to fulfilling the current intention. The intention generator Primitive 
behavior routines, such as obstacle avoidance, and more sophisticated motivational 
behavior routines, such as mating, are the building blocks of the behavioral repertoire 
of the artificial animal. 
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5 Interfaces 

Because it is our past experience that all technology much be bulletproof and 
withstand the rigorous demands of an unrelenting public, this equipment is designed 
along the lines of actual DIVE equipment which is very durable and rugged, big and 
easy to use. 



5.1 Tracking/Sensing 

A number of tracking options are available, his environment will consider the use 
of mix from magnetic [17], to un-encumbered technology such as camera based 
systems [18]. Sensing the bulk of visitors is done with infrared and video. Specific 
occupant tracking is done either with localized gloves cabled to the specific site 
location, or through wireless. Shutter glasses should be independent and able to sense 
any WALL or CAVE. Artificial Life use sensing to establish the location of visitors 
within the environment and can respond to them as a normal fish would respond. 



5.2 Weight Belt with Dive Panel 

This equipment enables visitors to interact with the environment and gain statistical 
information about the life species in the installation. The weight belt holds the battery 
pack and wireless communications, tracking and wand/pointer hardware, and a 
smart/memory card slot for recording individual data and experiences. Attached to it 
is a "Dive Panel" which is a pointing wand, and small LCD monitor to display 
information about the species that are under investigation, and other climatic 
information. Through this, the visitor is able to access the knowledge base stored on 
the memory card and other data being generated real-time. 



5.3 Enhanced Shutterglasses 

Shutter glasses are connected to a timer so that when the visitor's use all their "air", 
they shut down and are inoperable. This makes the environment very difficult to see 
and forces the visitor to exit the installation. Some of the more sophisticated models 
for avatars and "Marine Biologists" have audio communications built in. 



5.4 Taking Home a Bit of the Virtual Reef 

Something else that we think is really important from NICE is giving people some 
kind of artifact from the experience. Since VR is such a rare experience, and you 
really can't take anything physical away from the virtual space, it helps people when 
they try to describe what they did to other people, and enhance the experience and 
memory of the event. 

Each participant in the virtual reef environment gets a smart/memory card to record 
a "narrative" of everything they did, characters they talked to, actions they performed. 
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The narrative structure captures these interactions in the form of simple sentences 
such as: "Mary (disguised as a dolphin) stops by to see Murray the Moray eel to find 
out what he eats. Mary and Murray have dinner together." 

The story sequence goes through a simple parser, which replaces some of the 
words with their iconic representations and stores the transcript onto their Memory 
card. This gives the story a "picturebook" style that the visitor can print out and keep 
to remember the various aquatic life they met that day. 



5.5 Smart/Memory Cards 

We are currently investigating the use of Smart/Memory Cards to perform functions 
like: tracking and recording the user's actions, supplying a knowledge database, and 
providing financial calculations for visitation fees. 



6 Interactivity 



6.1 Visitor Interactivity 

Avatar: There are many species in which a visitor can explore, from fish and other 
animals. Avatars can be as simple as the ones found at the localized exhibits for 
tourists to try out, to the sophisticated ones like dolphins and sharks. Avatars 
ultimately come under the domain of the simulated eco-cycle, so anything could 
happen: the visitor's avatar could be eaten at any time, and eventually will go through 
the entire life cycle of the reef. 

Snorkeler: This level is simple roaming and exploring the environment with 

minimal hardware, such as shutter-glasses to immerse them in 3D. The smart card 
enables them to utilize the local interactive stands available throughout the 
environment. 

Marine Biologist: This level of interaction is highly independent and is able to 
program their own interactivity levels for simulation, planning, studying, etc. The 
equipment this visitor has is the most advanced versions of the standard models. 



6.2 Artificial Life Interactivity 

The artificial life in the environment is just as interactive towards visitors as visitors 
are to them. They can and will interact with visitors in many different behaviors, 
from being friendly, territorial, viscous, funny, etc. Fast movement will scatter fish. In 
this way, true interactivity is two-way, creating a complex environment. 
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6.3 Staff Interactivity 

Guides: These are sophisticated avatars that can interact with anyone in the 
environment. These people can be in the environment or hidden from view. They 
have audio and visual communication devices to drive avatars such as Mermaids, 
large fish, human animations or indigenous characters. This level of interactivity can 
be used for guided tours, assistants, observers and other special requirements. At this 
level, the guide is able to leave discovery objects which visitors can touch and find 
out more information. 



7 Conclusion and Future Works 

This system has been designed using existing and known solutions of virtual reality, 
networking, evolutional theory and artificial life, and yet, the innovation lies within 
the organization of each application into complexity. It has been designed with the 
forethought of being able to use the physical architecture as a generic shell to enable 
other stories besides the reef to be told, and the application able to be used in other 
systems, including retrofitted Planetariums. 
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Abstract. Surface mines are in operation world-wide, the vast majority employ 
large haul trucks for the transfer of material both to the outside world and 
around the site. The sheer size of these trucks and the operating conditions 
means there is a high level of risk. Allied to this, the commercial nature of the 
operation means that down time is extremely costly and driver training 
expensive. The AIMS Research Unit has developed a PC based system to 
improve driver training which is currently being developed into a commercial 
application. 

Scenarios are created by importing site specific data through industrial CAD 
systems, road systems are then added through an editor to create good replicas 
of the environment facing drivers on a day to day basis. The world is further 
enhanced by allowing the user to specify a number of intelligent objects 
including haulage trucks, excavators with load points and various static objects. 

Once scenarios have been created training is carried out on a full screen real 
time simulation which allows trainees to drive or be driven by computer 
through the world. At any given point the trainee is able to stop the simulation 
and identify potential hazards, their associated risk, and take possible corrective 
action. 

Keywords: Driving Simulators, Safety Training, Virtual Reality, Hazard 
Awareness, Surface Mining. 



AIMS Unit 

The AIMS Research Unit at the University of Nottingham has been involved in the 
fields of computer graphics and virtual reality for a number of years and has identified 
a number of potential applications for the engineering industry. In particular it now 
appears that the technologies have developed sufficiently and costs reduced to levels 
that mean even the smallest companies can seriously consider them. 
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The AIMS Unit uses a range of commercial packages and programming languages to 
create new computer based applications. Development takes place on both PC and 
Silicon Graphics platforms. However the unit has found that systems developed using 
PC technology are more easily ported to the industry due to reduced costs, the 
existing user base and the familiarity of PC technology [1]. 



Background 

There are many surface mines in operation throughout the world, all of these require 
the safe transfer of material around the site. In many cases this is achieved by haulage 
trucks, these large vehicles, often the size of a small house, have many associated 
safety issues. 

Their huge size together with the difficult environmental conditions introduces 
handling and visibility problems. There is a high level of risk associated with each 
truck, as the consequences of an accident can be extremely severe. Strict operational 
rules are enforced to minimise the potential for disaster, whilst these have been 
successful in reducing accidents, by 1995 industrial trucks were still the second 
leading cause of fatalities in the private sector behind only highway vehicle fatalities 
[2]. Indeed the OHSA found on average 107 fatalities and 38,330 injuries occurred 
annually in the workplace, furthermore it found present training standards to be 
ineffective in reducing the number of accidents involving powered industrial trucks. 

At the moment only trained operators can drive these trucks, periodic evaluation of 
each truck drivers performance is also required. Competency will depend on the 
ability of the vehicle operator to acquire, retain, and use the knowledge, skills and 
abilities that are necessary. 

Financially, haulage trucks are the largest cost item in the operating costs of a surface 
mine in some cases accounting for an estimated 50% of operating costs [3]. The large 
initial cost of these trucks also makes driver training expensive. 

Attitudes towards industrial safety have advanced significantly in recent years and the 
mining industry has not been an exception to this. The introduction of new legislation 
in the UK [4] has changed the emphasis of industrial law from prescriptive legislation 
into new management systems. 

Large mineral organisations are now continually looking for ways to improve their 
safety performance and techniques, which may help to reduce accident statistics, need 
to be investigated. The capacity to remember safety information from a three 
dimensional world is far greater than the ability to translate information from a 
printed page into a 'real' three-dimensional environment [5]. 
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AIMS Unit Truck Simulator 
System Details 

The system has been designed to run on a PC platform, prospective users identified 
this as particularly important due to the wide availability and low cost of the platform. 
The system used for training operates as a full screen application and provides both 
visual and 3D audio output. Trainees are able to interact with the system in a number 
of ways including through a custom built steering wheel and pedals. 

The system is split into two modules, the first is an editor allowing the trainer to 
configure and save a number of different scenarios, the second runs as a full screen 
simulation during training sessions. Both parts have been programmed in Visual C++ 
and DirectX is used as the graphics and sound API, this allows access to acceleration 
across a wide range of hardware. 

System performance varies according to the size of environment, however advances 
in graphical accelerators mean that high frame rates have been maintained despite 
using large pit meshes. The system has a high level of graphical detail, this is quite 
important during the process of training as it is often the ability to spot visual details 
that ultimately leads to safe operation. 



World Modelling 

As mining progresses from phase to phase, haul road networks and truck routes must 
adapt as the landform changes. To create more relevant training scenarios, it was 
important that the trainer should be able to accurately recreate the work place and 
conditions. This ranges from road layouts, truck movements and loading points 
through to lighting and fogging conditions to simulate different weather conditions. 

The application we have developed reduces the cost of creating new worlds by using 
currently available pit data. Surface mine operators maintain 3D models describing 
the development of the terrain and possibly haul road layout for each distinct 
operating phase. These are maintained in industry standard modelling packages such 
as Vulcan or Surpac. Mesh data held in these packages can be exported directly into 
the system and used as a base mesh for the virtual world. 
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Fig. 1. 2D pit during configuration 




Fig. 2. Construction of the haul road 
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Haul road information is overlaid using 2D drawing tools, it is then converted to a 2D 
road system, before being subtracted from the base model. Height values for the haul 
road are obtained through an averaging process on local vertices, smoothing also 
takes place on each segment of the road so that the surface remains flat. The road 
system is then re-triangulated back into the original mesh and textures are applied to 
both. This process creates a haul road system consisting of a number of straight, 
curved, and multiple exit junctions, this not only helps to reduce the polygon count 
but also to reduce the computational overhead during simulation. 




All further objects, their attributes and behaviours are added into the scenario at this 
stage by clicking on a 2D plan of the environment, the third height dimension is 
automatically calculated. The addition of objects and the configuration of their 
behaviours has been kept as simple as possible to reduce the complexity of defining 
new scenarios. 

Haulage trucks are added by indicating the start, finish and waypoints through which 
they must pass. Excavators are added by defining the segment in which they load and 
the direction from which trucks approach. Other objects, such as Surveyors, parked 
vehicles and road signs are added in plan view and their associated height values are 
again calculated. Further haul road system details can also be configured, this 
includes assigning priority at multiple exit junctions. 



Intelligent Behavioural Modelling 

Truck position and velocity is calculated in real-time and is based on industry 
standard rimpull and retarder curves, these together with factors such as mass, and 
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gradient provide an accurate simulation. Interaction with surrounding objects is also 
taken into account before updating a truck's position. 

Trucks attempt to follow user-defined routes based upon the road network that has 
been created, these are pre-calculated as far as possible by storing an ideal line which 
the trucks attempt to follow. Trucks will take a predefined route through the pit to the 
load point, where they queue to be loaded before proceeding to the dump point, and 
the cycle begins again. Trucks also have the intelligence to modify their position, 
velocity, and acceleration during the simulation according to the state of localised 
equipment. Figure 4 shows an example of multiple computer-controlled trucks 
queuing to be loaded. The simplification of the haul network allows extensive details 
as to the position of the truck within each segment to be monitored as well as 
allowing the truck to look ahead analyse potential situations and adjust its behaviour 
accordingly. 




Fig. 4. Trucks queuing to be loaded 



Training Methods 

Trainees may either drive or be driven (by the computer) around a pit. Training is 
primarily achieved through a technique termed 'hazard spotting', this allows the user 
to identify a number of pre-determined hazards which are included in the virtual 
world. As the trainee passes through the pit they are able to stop the simulation at any 
time to 'spot' these hazards. Once identified, they are asked to classify the hazard in 
terms of risk, and indicate any corrective action which should be taken. Performance 
is recorded and responses assessed against ideal behaviour. Figure 5 shows such a 
choice being made. 
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Fig. 5. Risk choice being made 



Future Work 



Work is currently being carried out to improve the current system and also to extend 
its scope. Proposed is a pre-start up equipment check allowing the trainee to inspect 
his vehicle prior to entering the simulation. Planned improvements for the current 
system include the use of multiple monitors to provide additional viewpoints and 
using aerial photographs as textures to overlay the pit. Development is also being 
carried out on using similar techniques in other industrial environments. 



Conclusions 

Virtual reality seems to be an ideal way of exposing trainees to hazardous situations 
without ever putting them at any real risk. Additionally a large number of potentially 
rare circumstances can be experienced in a relatively short period of time. However, 
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there is still doubt as to how well these skills transfer to the real world. Work with 
other vehicle simulators indicates that there can be a significant reduction in accidents 
[6] so it would be reasonable to expect similar results. 

The system has yet to receive field testing, however it has been demonstrated to a 
large number of mining institutions whose response has been very positive. 
Representatives particularly liked the fact that they could create and configure 
multiple scenarios from existing CAD data quickly and easily. It is anticipated that 
the system will be commercially available in the next few months. 
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Abstract. The aim of this research project is to utilise VR models in urban 
planning in order to provide easy-to-use visualisation tools that will allow non- 
experts to understand the implications of proposed changes to their city. In this 
paper, the navigation problems identified whilst working on large VR city 
models are discussed and a “fly” based navigation mode is proposed and 
evaluated. 



1 Background 



The Centre for Advanced Studies in Architecture (CASA) has been involved in 
three-dimensional (3D) computer modelling for the last six years. In 1991 CASA 
received a grant to construct a 3D computer model of Bath [2]. The project was 
supported hy Bath City Council and since its completion the model has been used by 
the city planners to test the visual impact of a number of proposed developments in 
the city. The model was created from aerial photographs of the city in 1:1250 scale 
using a stereo digitiser. It is accurate to 0.5 metre and covers the whole historic city 
centre, an approximate area of 2. 5x3.0 km. During 1995 and using similar techniques, 
the core of London’s West End covering 1.0X0.5km was also modelled followed by 
Gloucester city centre in 1997. 

Following the hardware and software developments of the last few years, the 
expansion of the Internet and the World Wide Web (WWW) and current trends in the 
industry, the Bath and London model were translated and are used non-immersively 
in VRML as well as custom made VR applications [2][3], The initial problem faced 
was how VR applications would scale and adopt to visualising a whole city of over 
three million triangles as in the case of Bath. The Bath database is, to the author’s 
knowledge, the largest and most detailed one produced as yet; the UCLA Dept, of 
Architecture and Urban Design (AUD) is currently building a model of the entire Los 
Angeles basin covering an area in excess of 10000 square miles as part of “The 
Virtual World Data Server Project” but it is still under construction. 
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2 Utilising VR Urban Models 

The computer models created in CASA demonstrate how computers will be used 
in the near future by engineers as part of their everyday practice, creating, modifying 
and improving our cities online using centrally stored sharable databases. The aim is 
not to create yet another Geographical Information System (GIS) although GIS 
features can be incorporated. Using Internet compatible programming languages such 
as Java, Cobra or ActiveX, VR urban models can replace a dedicated GIS system for 
certain applications (London’s West End). It should be noted that due to the nature of 
the proposed use of the models, the low polygon count fully texture mapped model 
approach adopted by more commercial/advertising oriented projects (Virtual Soma, 
Virtual LA etc.) was not feasible. 

The Bath model has been used in a variety of ways since it was originally 
constructed. To date, development control has been the main use with a number of 
schemes being considered. These are normally at the instigation of the local authority 
who recommend that schemes are modelled in order to facilitate discussions during 
the design phase and for presentation to the planning committee. In addition to its use 
in development control, the model has also been used to widen the public debate on 
how the city should develop in the future. 

The model of London’s West End is of lower level of detail and was initially used 
for transmitters' signal propagation experiments by British Telecom (BT). CASA has 
been using it as a database front end to navigating and mapping information on the 
city thus creating the Map of the Euture. Gloucester’s city centre model which was 
commissioned by the Gloucester City Council is used for planning control similarly 
to the Bath one. 



3 Navigation Problems 

Early in the process of creating VR models, the problem of navigation was 
identified together with inconsistencies on texture mapping, instancing, materials, 
indexing, etc. Although most of the problems were solved [9], navigation remains a 
troublesome experience not only to occasional users of the VR models but to the 
creators as well. 

The main problem in exploring, navigating and generally working with urban 
models in a VR environment (both immersive and non-immersive) is orienting 
oneself; being able to identify areas, streets, buildings etc. Therefore, a great deal of 
effort has been put towards making urban models more recognisable. It should be 
noted that there is a great difference between pursuing realism and aiming for a 
recognisable virtual environment (VE); a VE does not necessarily imitate reality [5]. 
Creating realistic VE of such scale is still not feasible and in many cases both 
pointless and inappropriate. 

In real life, when a person becomes disoriented in a urban environment, the natural 
action taken is to scan the surroundings searching for information. As Lynch [14] 
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explains, there is a consistent use and organisation of definite sensory cues from the 
external environment. 

Assuming no external help is requested, this activity comprises of three steps: 

• Look around 

• Move around within a small radius 

• Wander further away with the added complexity the person may subsequently fail 
to return to the original position. 

Failure of the above three steps means that more drastic ones must be taken. This 
typically involves interaction with other people asking shop owners, pedestrians, car 
drivers, etc. for the relevant information. Alternatively, one could check road names 
against a map, in extreme cases even consult a compass. 



3.1 Current Trends 

There have been many attempts, both commercial and theoretical, to create and 
interact with informational spaces [6]. Techniques have been developed based on 
both “realistic” and “unrealistic” concepts but it seems there is no consensus as yet. 
Putting aside the information space metaphors used, the navigation metaphors 
employed can be classified in two main categories; the screen and the world based 
ones. Among the former is sliding, rolling and examining whereas the latter include 
walking with or without physical constrains (gravity and collision detection), flying 
above the landscape and above the roof line utilising a bird’s eye view of the city and 
borrowing from cinematic terminology, panning and tilting. 

Game development is an area that one can review design decisions; a very 
efficient, demand driven, competitive environment enforcing quick, effective 
development cycles. 

Early 3D games like Doom and Descent, where the focus was on speed and 
interactivity, featured fairly simple environments. Each stage had an average of 10-20 
rooms/spaces with quite different shapes, textures and themes in general. 
Furthermore, the overall settings for these games were not real 3D; players could not 
share plan co-ordinates (XY) with different elevations. The cues facilitating 
navigation were plenty, with the different types of enemies and, in some games, even 
their corpses adding to the amount of visual information available. Auditory cues 
were vital in sensing events that took place in secluded areas or simply behind the 
players' back. In such games a 2D wireframe plan of the site was the only navigation 
hint given to the players and was usually enough to get them back in course. 

In the more recent “action” games like Tomb Raider and Tie Fighter, the following 
important differences can be identified: 

• The player’s avatar is fully included in the VE. The player is not a spaceship 
dashboard or a weapon carrying hand anymore but a fully articulated avatar giving 
better sense of scale and participation [11] 

• More advanced navigation aids are used (notably compasses, text narration, 
lighting, sound etc.) 

• Real-time rendering of texture mapped animated characters 
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• Collision detection and gravity 

• “Director’s” Camera Movement. Camera zooms and moves around the action (that 
is the player’s avatar) creating a feeling of a well-directed action movie rather than 
a simple game playing (Tombraider). Subsequently, wandering in the rooms 
becomes a worthwhile experience itself! 

The scale issue addressed with the full avatar presence is the most notable 
improvement together with the use of perspective and camera movement to follow the 
action. The above mentioned points can be and in a few cases are already employed 
in VR urban models’ interface. Information providing with position and orientation of 
the user visible as well as custom dashboards featuring compasses are possible. 
Additionally, audio data, animated objects can be incorporated. 



3.2 “Flying” Modes of Interaction 

In this paper, the focus is on the implications of employing a “flying” based 
navigation mode. It is argued that due to the lack of sufficient detail at street level, the 
“identity” and “structure” of the urban image is much “stronger” from an elevated 
position, when more distant cues are visible. This is especially true in the CASA built 
models considering the fact that the 3D models were created from mainly aerial 
information; facade surveys were carried out but the main core of information came 
from stereo pairs of aerial photographs. The roofline is very accurately represented in 
geometry and colour without the need for textures or complicated geometrical 
hierarchies. In many cases, roof level detail had to be eliminated in order to keep an 
overall balance within the VE. 

Among the problems linked to the “fly” mode is that it effectively removes any 
degree of immersion by switching the person to map reading mode (some will argue 
that it furthermore defies the reason of having a VE in the first place). Prior 
knowledge of the city is advantageous whereas there is a distinct lack of sense of time 
and effort needed to travel over a VE. 

Eurthermore, according to Ihde [12] there is evidence that there are important 
connections between the bodily-sensory perception, cultural perceptions and the use 
of maps as navigational aid. He relates the bird’s eye view of flying over a VE to the 
God’s eye map projection identified in the literary cultures. This mode of interaction 
has definite advantages although it also introduces an often-unknown perspective of 
the city. This perspective is more accessible to engineers and architects who are used 
to working with scale models of sites which inherently introduce the bird’s eye view. 
According to Tweed [18], the relationship between flying and walking modes should 
reflect to orientation of the body position, making flying modes more akin to 
immersive VR systems. 

It should be noted that it is possible to add street level detail in order to improve 
the amount of information -sensory cues available by adding: 

• Street furniture (lamp-posts, phone-boxes, bus stops, litter boxes, road signs, 

tarmac markings ...) 

• Building detail ( Textures + geometry) 
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• Animated elements (vehicles, people, etc.) 

• 3D Sound and pre-rendered shadows 

• Trees, landscaping, flowerbeds etc. 

However, more often than not, time and cost constrains prevail (surveying and 
modelling are both costly and time consuming) not to mention the hardware platform 
limitations [13]. Adding street furniture, landscaping and animated elements puts the 
burden on both hardware and software. Frame rate is an issue; from experiments 
carried out already in CASA, the indications are that 5-6Hz is the lowest acceptable 
level for such applications assuming the key issues addressed above are catered for. 
However, frame rate is not the most important variable in urban scale models — a 
30Hz VE of an inferior model lacking the key issues is not a satisfactory solution. 

It should be noted that urban VE rely very heavily on visual cues in many cases 
ignoring the fact that aural cues could be more powerful and helpful [7]. This can be 
credited to the fact that most such models are created and used by architects and 
engineers in general who are traditionally not taught to understand and appreciate the 
importance of sound in an environment, even more so in a VE. 



3.3 Limitations of Non-immersive VR 

VR models that are used in a team evaluation environment are usually non- 
immersive. The main reasons are practical; if all the committee members are to be 
gathered together in one room, the amount of hardware and cables running on the 
floor, the size of the room and the overall support required is unfeasible. The cost of 
the necessary hardware to provide a fully immersive experience for half a dozen 
planners and other committee members is prohibiting. Finally, the need for 
interaction and communication while evaluating a scheme and the need for a common 
reference point when assessing a particular feature would be extremely difficult to 
achieve in an immersive environment [1]. For such tasks, the Reality Centres that 
Silicon Graphics Inc. has created are ideal; pseudo-immersive double curvature, wide 
screen displays with a single operator. On the downside, the assessors' team has to 
travel to the particular site (only a handful in the whole UK) and the costs involved 
are outside the budget of a City Council or local authority. 

Nevertheless, there are important navigation advantages to be obtained on a single 
user immersive VE as opposed to a multi-user one. 

One of the problems identified in early experiments is the lack of concept of time 
and distance. Walking on the streets of a virtual city is an effortless exercise in 
contrast to the real experience. It is possible to fly over a whole city model in a matter 
of seconds and that can be instrumental on loosing both orientation and sense of 
presence. Recent attempts in immersive VE tried to introduce body interfaces to 
navigation. Shaw [16] used a bicycle as the metaphor for navigating in his installation 
The Legible City. This way, the pedalling speed sets the speed of movement whereas 
the handlebars determine the direction. Another metaphor used with movement 
tracking devices is that of walking on the spot [17]. The pattern of body movement 
relates to pre-computed patterns and determines whether the user walks, runs or steps 
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back. Char Davies has created a synthetic environment called “Osmosis” [8] 
exploring the relationship between exterior nature and interior self where the 
immersant’s body movements are triggering series of events that position the body 
within the VE and even alter the environment itself. 

Among the first techniques employed in the Bath model in order to improve 
sensory cues from the environment was the use of wide angle lens; the argument 
being that this way a greater part of the city is visible and thus more information is 
transmitted to the user. However, the results of a small scale case study with 
university students of architecture and other members of staff familiar with the city 
where discouraging. Architects identified the problem as being that of “wrong 
perspective” compressing the depth of the image and generally creating a “false” 
image of the city. Others could not easily identify the problem but found the 
environment confusing nevertheless. Consequently it was decided to use only the 
“normal” lens on all VR projects although it does limit the perceived field of view 
which in real life is much higher than 45 degrees. Experiments on level of detail 
degradation in the periphery of head mounted displays (HMD) [19] as well as eye 
movement and feedback [4] demonstrate that more efficient VR interfaces can be 
achieved (compared to non-immersive ones) without necessarily hitting on the main 
VR problem; CPU capabilities. 

Another problem faced in non-immersive VR is that of the direction of movement 
versus direction of sight. Due to the two dimensionality of the majority of input 
devices used in non-immersive VR, it is assumed that the user looks at the exact 
direction of the movement. This is quite true in most cases, but when one learns and 
investigates a new environment, movement and direction of viewing should be dealt 
as two individual variables. Immersive VR headsets with position and orientation 
tracking mechanisms are again the easiest and more intuitive solution to this problem. 

Therefore the Variable Height Navigation Mode (VHNM) is introduced as another 
solution to the problem. 



4 Variable Height Navigation Mode 

The VHNM is based on the fact that at any given position in a urban model, there 
must be a minimum amount of information - sensory cues - from the external 
environment available to assist navigation. This can be achieved by adding enough 
information on the model although this is rarely an option as was discussed earlier. 
Alternatively it can be achieved by varying the height of navigation according to the 
amount of sensory cues available on any given position. 

As an example, a wide long street with a few landmarks will bring the avatar down 
close to street level, whereas a narrow twisted street in a housing area will force the 
avatar high above the roof level. 
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4.1 Theoretical Model 

In the real world, the sun, the wind, a river, the sound emitted from a market or a 
busy street, a church, a building, a street, a road sign, a set of traffic lights are among 
the sensory cues accessible and used by people. Furthermore, each person gives 
different importance to each of them making the whole process of classifying and 
evaluating visual cues much more difficult. In a VE, which is generally much less 
densely occupied and furnished by the above-described elements, classification is 
relatively easier. Although the user must be able to rank the relative importance of the 
various cues available, the designer of the VE can assign with relative safety what is a 
sensory cue within the VE. 

The theoretical model proposed uses a series of gravity like nodes or attractors [7] 
that pull the avatar towards the ground when approaching an area of high density in 
sensory cues. Similarly, in low-density areas a global negative gravity pulls the avatar 
towards the sky until there are enough sensory cues nodes visible to create 
equilibrium. It should be noted that the attractors are not true gravity nodes since 
they can only affect the elevation of the avatar and not its position in plan. 

The first emerging issue is that of cues’ visibility. In order to identify the visual 
cues available from each position the field of view and the direction of viewing is 
considered. Following, a ray-tracing algorithm is employed in order to assess visual 
contact from the current position. Considering the size of the object, a series of rays 
must be calculated to establish the percentage visible from the current position. 

Relative importance of the visual cues is essential if the model is to be successful. 
A general ranking of them can be carried out by the world designer based on the size, 
form, prominence of spatial location, historic value, users awareness, etc. However, 
the navigation mode proposed should be flexible enough to accommodate particular 
needs of the users. Locals will use completely different cues (depending on the 
culture, could be pubs, churches, prominent buildings, etc.) to the first time visitors 
who will probably opt for the ones described in their guide books and the ones that 
possess the main landmark characteristics as defined by Lynch [14]. 

Another variable relates to the amount of visual cues users received over time 
while navigating in the VE. The higher the amount the better the mental image of the 
city they have created. According to Lynch [14], sequential series of landmarks where 
key details trigger specific moves of the visitor is the standard way that people travel 
through a city (p.83). VHNM keeps track of the number of visible cues, their distance 
from the users’ path the speed and direction of movement and finally the time each 
one was visible. All that can recreate the path followed and should enable the 
prediction of areas of potential problems and modify the current elevation 
accordingly. In the event of orientation loss, it would be possible to animate back to a 
position rich in visual cues, a kind of an 'undo' option. The main question in 
animating back to a recognisable position is whether the user should be dragged 
backwards (as in a process exactly reverse to the one they followed) or if the viewing 
direction should be inverted so that a new perspective of the city is recorded. The 
argument for the former is that the user will be able to directly relate to the actions 
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taken only minutes ago, whereas the latter has the advantage of showing a new view 
of the city and the disadvantage that the process of memorisation may break down. 

Consequently, the VR application can be trained over a period of time and 
eventually be able to react and adjust to the habits and patterns followed by each user. 
Keeping track of the time spend on particular areas of the VE will enable the 
computer adjust the avatar’s position on the assumption that the recognition process 
and memorisation will strengthen the more time one spends at a particular place 
within the model. 

The tilt of the viewing should be also briefly considered. Walking at ground level 
and looking straight ahead is what people are used to, but when one is elevated 
twenty, thirty or more metres above the ground level the perspective distortion 
introduced by tilting and looking downwards should be considered. As described 
earlier, an often unknown perspective of the city is introduced and the effects it may 
have on different users should be carefully evaluated. 

Concluding, the theoretical VHNM model proposed will be quite difficult to 
implement in real time with existing computing power in large VR urban models. It is 
currently an extremely difficult task to structure the geometric database alone; setting 
up all the raytracing calculations and the fairly complex set of rules described above 
will seriously affect the performance of any VR application. However, some 
problems may be alleviated by reorganising the scenegraph, which conflicts with the 
general concept of spatial subdivision of large VR models and their organisation in 
Levels of Detail as discussed extensively elsewhere [2], [3]. 
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4.2 Model Implemented 

Having described the ideal system an attempt is made to put it into practice. 
Bearing in mind the difficulties in implementing the original concept, a collision 
detection based navigation mode is initially proposed (Fig. 1). A transparent and thus 
invisible mesh is introduced, “floating” above the urban model. The peaks of this 
mesh are where the minimum amount of sensory cues are, so in a way it is an inverted 
3D plot of the sensory cues against the 2D urban plan. Using a 2D device, such as a 
mouse, it is possible to navigate in the VE whilst collision detection against the 
invisible mesh determines the elevation of the viewer (Fig. 2). 

b. 



Fig. 2. VHNM views on a. high and b. low visual cue areas 

Early in the development stage, it was decided to represent the VHNM mesh as a 
visible five-metre wireframe grid, giving users a reference point to the height they are 
and an extra orientation directional grid. Consequently, the VHNM mesh clearly 
denotes which areas of the model are lacking sensory cues and by creating visible 
peaks on them, it actually presents cues of its own to aid orientation and exploration. 
Eollowing extensive testing, it was concluded that for the Bath model, areas rich in 
sensory cues can be successfully explored at a height of 25 to 35 metres above street 
level (10 to 15 metres above the buildings' roofline). Regarding areas that proved 
problematic in the initial navigation experiments, 40 to 50 metres above street level 
produced good results. It was attempted to keep the height variation to distance ratio 
as low as possible to avoid user confusion and make navigation and movement 
smoother. It should be noted that the values mentioned above were satisfactory in the 
particular urban model but it is unlikely they will be suitable for a high-rise or a very 
densely built area. Experimentation will provide the right values and most likely a 
mathematical model will be developed once the VHNM is tested on London's West 
End and Gloucester City centre models also developed in CASA. 

In order to accommodate for the different needs of varying groups of users, the 
VHNM mesh can be shifted along the height axis (Z) according to the familiarity of 
the user to the urban model presented (Eigure 3c,d) using on screen controls. It is 
also possible to alter the scale of the mesh along the Z-axis. Doing so, the viewer 
drops lower closer to the ground on areas rich in sensory cues and flies higher on 
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poor areas (Figure 3a, b). The first experiments carried out using the manually created 
VHNM meshes were quite successful in reducing the amount of confusion and 
orientation loss. However, no statistical analysis of the data obtained has been carried 
out yet, since controlling such an experiment proved extremely difficult. 
Consequently, most of experimentation was used for fine-tuning the technique and 
the variables involved. A series of tasks are currently being developed in order to 
create a more controlled environment enabling drawing of conclusions on the 
effectiveness of the proposed method and creating a test bed for evaluation of future 
work. 




Fig. 3. VHNM Editing options, a. normal, b. scaled up, c. shifted down, d. shifted up 

Among the limitations of this implementation is that the computationally 
expensive ray-tracing calculations needed to accurately create the mesh, were 
approximated in advance manually. Consequently, the already complicated and 
difficult job of the VR application is not stressed any further by having to compute 
gravity nodes and search for cues in the VE. However, generating this mesh is a very 
laborious process and difficult to automate. Furthermore, the VHNM mesh 
completely disregards the avatar’s viewing direction. In most positions within an 
urban model, the cues available are strongly related to the viewing direction. Looking 
north may give no clues whatsoever whereas turning 180 degrees and facing south 
may reveal the position of the sun (at least in the north hemisphere) and a landmark 






Navigation in Large VR Urban Models 



355 



that will let you pinpoint your position on a map. Users have no option to classify the 
relative importance of the type of cues available according to their own needs. It is a 
pre-calculated VE that one is only allowed to navigate in, which is conflicting with 
the general conceptions of the service computers should be providing [15]. 

The next step will be to incorporate the visual cues versus time variable in the 
model and investigate to what extend the behaviour of a visitor can be predicted and 
augmented. Following, auditory cues will be included while work will be carried out 
into finding ways to simulate more accurately the theoretical model proposed. 



5 Conclusions 

The proposed VHNM can solve certain problems in urban VR environments and if 
implemented fully could be a very successful navigational tool. However, bearing in 
mind the complexity of the proposed model, an immersive VR environment would 
supplement a partial implementation of VHNM more economically, in terms of cost, 
ease of programming and time needed to construct and fine tune the VR system. The 
implementation problems of the proposed VHNM can be tackled with different 
degrees of complexity and accuracy according to the capabilities of the VR platform 
used. 
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Today's world is characterized by an explosion of new communication 
technologies and unique uses which modify our behaviour and our thought 
processes. Confronted with this evolution, we are waiting for a new form of life to 
emerge corresponding to these technologies. A new art form corresponds to this 
existence, not merely by tools that it uses but through the questions that it raises. 

We can thus note that with the intervening changes in society, scientific 
progress, and the change in the general relation between art and life, the artist's role 
today is radically new. Contemporary art has changed its function in relation to 
modern art in the sense that it resembles more and more a science. 

Since its conception, art has appeared as a representation of a life ideal, 
relative to one time. Contemporary art doesn't reflect life. It prefigures it. At present , 
thanks to the new digital technologies and communication,art itself can become a 
simulation of reality: an immaterial digital space which we can literally pentrate 
and,as Fred Forest says, is no longer , a " self-defining form . It is itself a picture, 
formalized numerically. Dimensional space is no longer an intangible substratum. It is 
a digital object in interaction with other created objects , having its own reality". 

Art exists thus as the counterweight and the complement of science, so that 
the two together complete the human discovery process. The artist creates, therefore, 
from concept to analysis while the scientist works from analysis to concept. The artist 
elaborates a concept as a starting point from which he participates in the construction 
of a world. If he succeeds in proving the reality of his concept, it will develop to 
become part of the collective conscience. On the contrary, the scientist analyzes, 
researches, studies, and experiments before formulating his concept. In this way, 
science reveals our knowledge of the world , exploring it step by step, while art 
interests itself with experience of the world,offering us concepts that permit us to 
glimpse immediately another, more intuitive, vision of the universe. 

Finally, the questions that artists ask themselves in their research are 
extremely close of those of scientists. Communication artists and researchers, 
working on the problems of communication are confronted, according to Pierre 
Moeglin, by the same question: In our modern age, to what do we owe this central 
position ceded to video, to the computer and to telecommunications? Certainly not, 
they answer, to the sophistication of these technologies, installing a difference of 
technological application but not of its inherent nature in relation to the previous 
techniques. Furthermore, we can say that while pushing their limits to phenomena 
that are not all new - several were already germinating during the first industrial 
revolution - uses are made systematically revealing better than before the wider 
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dimension of diffusion that includes our relation to others and ourselves as well as the 
access that we have to work, to leisure and culture . 

The vocation of art, as well as that of science, has changed. Art's role is no 
longer to entertain but to communicate something about the nature of the human 
mind: the scientist teaches from his scientific paradigm, the artist from his vision of 
the world. 

The different and unique nature of cyberspace permits us to simultaneously 
approach and escape from reality. Today we are not content with computer-driven 
animation but moreso with attempting to recreate life, to breathe independant life into 
these " beings " of pixels and algorithms. It is about giving birth to these " virtual " 
creatures, capable of learning and evolving. 

Virtual reality in relation to the real world implies an isolation. It seems the 
most obvious expression of a man / machine system in which man plays an integral 
part. In this universe, movements, or even physiological data and human brainwaves, 
can be controled by sensory stimulations used like commands. It is a device placed in 
contact with our body, in our senses, on and maybe one day even under our skin, 
directly in our brain, if one succeeds in developing brainchips. Some researchers 
even speak about associating the computer system directly to the human brain. Our 
senses will then be pacemakers that activate our brain by their signals transformed 
into electric activity. 

More and more realistic "clones", shall circulate throughout the network, 
with all manners of functionality and with an impressive delegation of power 
devolving directly from their "game-masters ". The mixture of reality and virtuality 
applies directly to the creation of cloning. Thus, the putting into movement of virtual 
actors in real time can be of use in as different domains as teleworking, 
videoconferences, virtual communities,multiparticipant games or 3D animation. 
These actors are derived from real faces, then sculpted in 3D before finally being 
cloned. The animation of clones is produced then in real time: the face of the user 
drives directly his duplicate with the help of a camera and a treatment of images 
while the top of the body is directed with the help of two sensors, one on each hand. 
The user can communicate thus with other clones, to submerge and to interact within 
a 3D universe. 

At the same time, avatars install in the network a new aesthetic that would be 
the one of Tex Avery with a touch of Tolkien and a personality like the Simpsons. 
Masters identify more and more with their virtual image, so that avatars impose 
themselves in their daily life and transform their minds little by little as well astheir 
own body. 

The MUD and the MOO (carrying trades and games in 3D), that are 
developed in the same way as Internet, evolved in the same sense. Their virtual 
reality, founded first exclusively on the text, moved toward the conviviality and 
exchanges ludiqueses with representations in 2D and 3D of users nominees as of 
avatars. 
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Artificial comedians also became the object of various researches, in the 
scientific laboratories and in artistics studio. Interactivity in real time with these 
comedians became possible with the tele-animation of characters from movements 
and the mimic of real actors equipped with datasuits and sensors. 

Currently, the technical progress, the rythm of life and especially the new 
technologies of communication, are changing the body. The body is replacing by 
prostheses. the usefulness of its senses, sometimes leaving only one activated, 
deconstructed it. The body is seized and transposed in the virtual world, it is cloned 
and finally it is substituted by a new virtual body, the avatar. The body loses its 
consistence therefore, its substance, its value, and even its biologic role; it turns into 
interfacing. 

The carnal art, created by Orlan, interrogates the statute of the body at the 
genetic manipulation age, as well as its future that turns to a gosthly appearance . The 
virtual, to Orlan, does not only consist to multiply the body and to be telepresent, but 
also to accept the reconfiguration and the sculpting of the body. For her seventh 
operation- surgical-performance, implants, normally used to heighten cheekbones, 
were placed on every side of her forehead . It created two bumps. These bumps, being 
totally virtual, “avatar ized” the body of Orlan. 

As one saw it, the imitation of gestures and even of some human senses is 
today easy. But the virtual makes also possible the control of senses. Indeed, it 
reintroduces a kind of individual sovereignty, even though it is rather illusive. Of 
course, our sensory perception is necessarily coded, because of our cultural, social or 
personal history. Nor uncertain nor genetic, it is the result of the modelling to which 
all society proceeds. However, in the virtual reality, this modelling proves to be more 
restricting, more precise. This testifies well that sensory control is more there a 
fantasy. Therefore in a reasoning: “The sovereignty of the individual is maybe then 
restored. But at the same time, his liberty is forced infinitely, since, being not 
anymore a prey to confusions of the world or of the meetings, he cannot feed himself 
anymore with their wealth”. 




Las Meninas in VR: Storytelling and the Illusion 

in Art 



IS m iz 1 



w o nson 



istin 



sil kis 



Electronic Visualization Laboratory 
University of Illinois at Chicago, Chicago, IL 60607, USA 
(312) 996-3002 voice, (312) 413-7585 fax 
bizri@evl.uic . edu, http: //www. evl .uic . edu/ chris/meninas/ 



Abstract. Las Meninas is a virtual reality (VR) artwork based on the 
painting of the same name by Spanish painter Diego Velazquez. Created 
for the CAVE(tm), Las Meninas attempts to establish a language of art 
in virtual reality by placing VR in the realm of storytelling; storytelling 
that is not simply formalistic and decorative, but also psychological. 
The viewer confronts a narrative cryptogram which can be deciphered 
at multiple levels of meaning as he seeks to explore the enigmas inherent 
in the painting by literally entering into it, both physically and psycho- 
logically. This allows for the suspension of disbelief or the illusion in art, 
the quintessential rule of art. 
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oun t stu io n s t s n f om ny point in t 



oom. 





Las Meninas in VR: Storytelling and the Illusion in Art 



365 



vi w n t n ings im into t s n llowing im to s t s n s 
1 zqu z woul V s n it. is is t vi w s initi 1 nt n into t wo 1 
p viously un tt in 1 . p o n tion info ms t vi w t t 

is now f to xplo t is wo 1 . oving out t stu io t vi w no 

s V t s n f om ny point in t oom o f om ny t s point of 

vi w. Looking out t win ows of t stu io t vi w s s two non u li n 

sp s of volving p n Is. st s ows gu s w o influ n 1 zqu z s 

tim ( on s t s lil o v nt s n ot s). s on s ows t 
p s nt tion of m i g in p inting su s n n y s Arnolofini Mar- 
riage w i s p t w y 1 zqu z t oug t of t us of mi o s n fl 

tions to p s nt t visi 1 wo 1 6 . 



L ving t stu io vi t oo w y os Ni to st n s in t vi w nt s 

o i o wit p intings y zo 1 zqu z s pp nti ft u ns nging 
on t w 11s. is o i o 1 s to tow . oing n li n tion t t 
1 zqu z ss to tow f om wi sosvt vnst 

vi w lim s up t st ps of t is tow omp ni y musi f om s St. 
Matthew Passion. nt s oom wit t 1 s op s t im nsion 1 1 ipty 
n 1 g p inting of ist y 1 zqu z ims If s s own on t 1 ft in gu 

. L ving t is oom t y nt p ss g w y wit p intings y i sso n 
t ussi n p int v v ft Las Meninas . s t vi w mov s f om 

t 17t . to t 20t . ntu y t oug t is t nsition 1 o i o t musi s ifts 

f om t t of s fugu s n p lu s to t t of Lig ti n nittk s s own 

on t ig t in gu 




Fig. 5. n t 1 ft s 1 zqu z tow fo o s ving t v ns t 

vi w n lim vi tu 1 tow of t i own to t is o s v tion 1 oom. 

n t ig t t vi w w Iks f om t 17t . to t 20t . ntu y own llw y 
lin up wit p intings y i sso n t ussi n p int v v ft Las 

Meninas. 
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n tiv s iption of t p inting in lu s quot f om i sso t t 

Las Meninas sugg st to im t nt n of f s ist sol i s into t stu io of 

1 zqu z wit w nt fo is st. i sso s int st in Las Meninas s to 

0 wit its nt 1 1 m of t p int n is 1 tions ip to is mo Is n 

Iso wit its p ofoun m it tion upon t isto i 1 n so i t 1 p on ition 

of tisti tivity n pow 1 tions. n w sp t vi w ns ims If 

in s s V 11m nts pi ying on t is t m of pow n omin tion s s own 

in gu 6. 1 vision s ts nging in mi i juxt pos iv 1 Im foot g 

f om 1936 of n 1 n o of itl in Triumph des Willens n of plin 

in The Great Dictator. L g two im nsion 1 im g s of no n post s f om 
t p nis ivil o n t w 11s in f ont of mu Is of i sso s Guernica 

u ning oun ns s w Ik into t is s n f om out of on of 

1 sso s stu y p intings ft Las Meninas. 

in lly t vi w tu ns to t stu io s ing t s n f om t p sp 
tiv of os Ni to t t k of t oom. v 1 possi 1 p intings s own 
on t 1 nk nv s s in gu 7. tu n to t initi 1 sp is m k 

wit n w qu stions n n y t multi 1 y P sp tiv s t jou n y 

s V 1 to t vi w . Lif t t ou t of 1 zqu z w s st i tly i i 
1. omposition its If p s v s t is i y n m gin liz tion of t 

p int . s 1 zqu z t ying to correct t is i y t oug 11 go i In 
sym oli llusions out t n tu of p s nt tion n su j tivity? t 
wsitt t wsp inting on t i n nv s? s 1 zqu z p inting t 

King n t u n t Las Meninas its If o gu s of v y y lif n t us 

w s lling g inst t King n t u n n t nti t ition of ou t 
p inting? s 1 tions ips 11 s utiniz in t m ium of 

4 The Illusion in Art in VR 

om t outs t t vi w s ons ious n un ons ious min is t wo k m king 
inf n s n ing t n tiv yptog ms in Las Meninas. is ing 

is possi 1 us t vi w believes in w t s s n n i ntify wit 
it. t is in w t is 11 ontological authenticity t t t st ul of t 

illusion in tot ility to susp n is li f m nif sts its If. o t illusion 
in t to om su i nt on ition in k/s stimul tion of t s nso y 
moto s m is qui . ot 1 m nts ful 11 1 qui m nts fo i ving t 
illusion in t s w 11 s fo m listi lly n n ing t m ning of t t m ti 

n tiv plot. 

ntologi 1 ut nti ity f s to t illusionisti ility of t t im n 

sion 1 im g s to s ow n authentic p s nt tion of lity. v n if t s im g s 

unf mili to t um n y t y v to xt pol t f om t known 
in o to i V onvin ing p s nt tion. kin st ti imp t in 
is suit of t vi w s ontinuous n vig tion n t mov m nt of im g 
its If. syn st ti imp t is suit of t s ml ss int tion tw n t 

u ito y n t visu 11m nts. 
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Fig. 6. i sso s w issu s of pow n omin tion in Las Meninas n 
t vi w n xp i n 20t . ntu y int p t tion. 



Las Meninas m nif sts ontologi 1 ut nti ity n k/s stimul tion in m ny 
w ys. vi w is not only imm s in t t im nsion 1 im g s ut 
Iso believes in t real im g s of n v t stu io tow p ss g w ys n 

V ions isto i 1 n politi 1 ooms. u t mo t mis n s n lo tion 

soun n t us of t musi of Lig ti n nittk o st t 

in o to i V genuine realism n n n t motion Ip ti ip tion of 

t vi w in t t of V 1 tion. 

o st tion of visu In u ito y 1 m nts oft n st t n 
m unf mili in o to int nsify n stimul tt viw smntlp 

ti ip tion. st tion in Las Meninas is Iw ys xt n o xt pol t 

f om t known. fo t vi w is oft n imm s in t t im n 

sion 1 im g s n soun s w i estranged i. . on t on n t y pp 

n soun concrete n real ut t t s m tim t vi w knows t y 

synt ti n st ti lly oust u t . vi w s no illusion t t s is 
onf ont wit real im g y t s li v s in it us it is xt pol t 
f om t known. 

ntologi 1 ut nti ity is t fo 1 y upon w i k/s stimul tion is 

uilt to i V t illusion in t. n Las Meninas v yt ing out t jou n y 

is t st sig t concrete in o to w t vi w into t n tiv t oug 

t us of opti lly g n t im g s. s t sto y p o s ow v t in 

t g tion/int tion of long n vig tion lu n llu in to y 1 n s p s 
non ulinsps It n It motion su ns ifts f om 
olo to 1 k n w it t mix of 17t ntu y ton 1 musi n 20t ntu y 

ton 1 musi iz n isto t o 1 y int in st u tu n p oli 
styl t mpo 1 p ssu n sp ti 1 is ontinuiti s n t us of iv 1 Im 
foot g in vi tu 1 nvi onm nt t ognitiv imp t in t vi w . is 



imp 


t dramatizes t 
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tiv 
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on i 
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vi w 
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tion plot. 
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Fig. 7. tu ning k to t stu iow gnt viwsswt oul 

on t nv s. om t is n w p sp tiv looking k s v 1 possi iliti s 
n s n. 



in t is sp t t t SUSP nsion of is li f n t illusion in to u s. t is 

t t st ti s n psy ology om int twin 

n t following s tions w will 1 o t on ow ontologi 1 ut nti ity 
n k/s stimul tion in Las Meninas st lis language of art in 

4.1 Fusion of Optical and Virtual Images 

Las Meninas st ts w n t oo of 1 zqu z s stu io t im n 

sion 1 omput g n t im g op ns to 1 t n opti lly p o u v t 
pi ying t ol of t p int ims If nt t mpty sp w i is omput 

g n t . om t st t t vi w xp i n s narrative tension ising 

f om n imm i t os ill tion tw n t wo 1 of t real opti s n t t 

of t imaginary vi tu 1 nvi onm nt w i is nont 1 ss ontologi lly ut n 
ti . fo ontologi 1 ut nti ity is ou ly p s nt t oug opti s s 
w 11 s t oug vi tu 1 wo 1 m ut nti . is n tiv t nsion is 

t iz y t vi w s ility to see som t ing s ot 1 n im gin y 
simult n ously. livst tw t ss longs to t 1 ws of opti s ut 

t t s m tim xisting wit in t 1 ws of vi tu 1 nvi onm nt. 

n lin tw n t p s nt tion Int vi tu 1 fo s t vi w 
to fl t upon t im gin y sp mpi i lly. vi w nnot s y to ims If 
t t not ing is li v 1 in t is n tiv us it is vi tu 1 n non 1. 

opti 1 im g s 11 t g p mpi i lly s w 11 s t ontonologi lly ut nti 
vi tu 1 nvi onm nt. is m k s t vi w t 1 st initi lly identify wit t 
n tiv unfol ing. f ou s t is p ti is not n w in t isto y of t. ot 
t ists n t u lists mploy su t niqu s quit su ssfully in 

p inting n inmw t Iwsm estranged. 

not inst nt in Las Meninas w t fusion of t opti Int vi tu 1 

t k pi is tow s t n of t n tiv . ft t vi w 1 v s t wo 1 

of t 17t ntu y 1 zqu z n t fugu s of . . nt s t wo 1 

of t 20t ntu y to witn ss stu i s y i sso of Las Meninas is Guernica 
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t s i 1 musi of Lig ti n nittk ut ov 11 t vi w n ount s 
t 1 vision s ts susp n in mi i . s ts s ow iv 1 Im foot g of itl 
n n o n of plin. g in t in lusion of opti lly g n t 

im g s wit ontologi lly ut nti vi tu 1 nvi onm nt fun tions t m t 
t m ti 1 V 1 to p ovok fl tions on t nging m t o s of p s nt tion 
n su j tivity. vi w is onst ntly s ifting f om t p s nt tion 1 
to t vi tu 1 n k. t is in t is os ill tion t t t fo most 1 m nt of 
sto yt lling m nif sts its If t susp nsion of is li f o t illusion in t. 



4.2 Visual and Aural Guides 
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n Las Meninas, t ns i o gui 


ful 11s 


ou 1 fun tion. 


n vig t s 
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vi w t 


oug out t st of t 


n tiv 


n n t s n 


som 
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s w 11 p op ty possi 1 us of t so i 1 n tu of in t 

fo t ontinuous n vig tion n s ml ss int tion/int g tion 
tw n t visu In u ito y 1 m nts t in t vi w k/s stimul tion 
w i ws im ot psy ologi lly n p ysiologi lly into t tion plot. 

4.3 A Total Environment and the Double Articulation of Time 

ft t opti 1 1 zqu z t k s is pi in t mpty vi tu 1 stu io t vi w 

paints t st of t p inting t nf nt g it n ntou g using 

t w n in t lik p int us . t is wit su int tivity t t t 

vi w is 1 to t In tw n w t is p s nt in f ont of im t 
p nom non n is own m nipul tion of it in . is ou 1 ti ul tion of 

tim t t is tim t t 1 y xists in t p nom non n its m nipul tion 

y t vi w giv s t vi w t f ling t 1 1 lity p s nt in t 

is not only p s nt tion 1 ut Iso ontologi In su j tiv . vi w is 
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n lly 1 to p t of t p nom non. is n xt nsion of vi tu 1 wo 1 
in w i n s p n t min t out om of v nts. p nom non 

p ssing in tim n now int upt It It mov ot 

kw s n fo w s n ompl t o 1 ft s is. vi tu 1 nvi onm nt 
om s total environment in w i t vi w is ot n xt nsion n 

t mining f to of t nvi onm nt. 

vi w s p ti ip tion is not it y in Las Meninas. f oos s 
t in signs out t n tiv fu t isto i 1 politi In st ti signs 

V 1 to im w i p op Is t n tiv in t in i tion. vi w in 

volv int tivity is int insi to t n tiv s ov 11 st u tu n n ss y 

fo t unfol ing of t nigm s of p s nt tion. 

4.4 Dramatic Shift from the Formalistic to the Psychological 

Las Meninas is st g in su w y t t t is m ti s ift f om t fo 
m listi to t psy ologi 1. Not only o s t wo k inv nt p ss g w ys tow s 
t im nsion 1 t ipty s non u li n sp s t 1 s op s t nsp nt su 

f s n t 1 vision s ts w i ontologi lly nt nti ut it Iso p ovi s 

t m wit isto y in o to onn t t m wit t n tiv n giv t m 

m ning. vi w s t oi to n vig t n int t wit v ions isto 
i Ip io s f om t 17t to t 20t ntu ywi moyspi stsn 

musi fl ting t i isto i 1 politi In st ti sp i iti s. s ift 
f om on p io to not is n tt mpt to m k onn tions tw n v ions 

p io s n monst t ow fo m n fun tion t s on . psy ologi 1 
f to pi ys m jo ol . Not only o s t vi w xp i n sp i 

sensation ising f om t sp i fo m listi s t n musi ut Iso m nt 1 

t of p ption om s s pu ly on un ons ious inf n s s m k s s 

s n vig t s n int ts wit v ious s ts in i nt p io s. 

5 Conclusions 

llusion in t is ompl x topi n s its own limit tions n 

p igms w n n ing lity. n w look t gypti n t fo x mpl 

w it s illi nt sign ling syst m of o n not s lit 1 p s n 

t tion of lity. ut is t is t w y t gypti ns t ms Iv s s w t i t? 

ks t t t ton o fo mo ling in lig t n s w i 
m ins fun m nt 1 to 11 1 t v lopm nt of st n t. s in ito s of 
t 1 1 ition n inv nto s of two ks it is impo t nt to inv nt language 
wi nst wyounw tools of p o u tion op t n s p t futu 
of t. n it s ms t t f w wo ks t y to syst mi lly fo mul t n tisti 
position n n w ys to t 1 ngu g of t w os ss nti 1 fun tion is t 

m nif st tion of t illusion in t. 

n V lop syst m of s m t in w i t illusion in t is possi 1 

onsists of two 1 m nts ontologi 1 ut nti ity n k/s stimul tion of t 
s nso y moto s m . vi w s positiv spons ft Las Meninas w s 
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s own t t nt n tion 1 o i ty o t It oni ts n t ink u st 
1997 t 11s us t 1 1 y ompl t ly fo got out t t nology oft n 

t tty believed in t n tiv unfol ing n w psy ologi 1 involv in 

its t nsfo m tions. 

t st t vi w s xp i n t t ill of p f t illusion in t 

us t i g tw n t p nom non n t vi tu 1 is ok n. ow v 

on t is illusion w so it is ss nti 1 1 t som t ing Is 11s t g p us 

w w nt n xp t mo . isto y of t nologi lly s t is full of 

su inst n s. ly in m fo x mpl w s t ill us of t k n 

t t fli k ing im g s mot pi s n so fo t . L t u i n s w nt 

mo n it w s t oug m ti n tiv plots w t lin o non lin 
t t in m ti t v lop 

is f ing simil 11 ng to t t of its ousin t in m . it out 

ontologi 1 nt nti ity n k/simp t Las Meninas wonl v n n x is 

in ts n not wo k w illusion m nif sts its If s w t f 1 n 
t ink in f ont of t ytog ms of its vi tu 1 wo 1 . n ot wo s in Las 

Meninas t vi w not only witn ss s t f it ful n onvin ing p s nt tion 

of visu 1 xp i n t oug ontologi 1 nt nti ity nt Iso t f it ful 

oust u tion no st tion of 1 tion 1 mo 1 in w i t int pi y of 
im g n soun t igg in t vi w k/s stimul tion to ing out second 

reality. is s on lity o igin t s in t vi w s ons ions n un ons ions 

reaction to t vi tu 1 wo 1 n not in t vi tu 1 wo 1 its If. ilusion 

in art n lly m nif sts its If in t vi w s reaction to t vi tu 1 wo Ity 

xp i n . 
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Abstract. v n s in hnology h v m i possi lor vs 
ri h n r hi ur lly in ri vir u 1 worl s. h Mitologies pro 
is n mp o u iliz his hnology s m ns o r is i xpr s- 

sion n or h xplor ion o his ori 1 poli i 1 musi 1 n visu 1 

n rr iv s. Mitologies r ws inspir ion rom 1 rg pool o li r ry n 
r is i sour s y p uring h ir in r wining r 1 ionships in in - 
m i orm h n m king onn ions o h s rong n rr iv r i ion 

oohrm i suhs film n li r ur . 

1 Introduction 

V n e in te nolo y ve m e it po i le to re te v t ri n intri te 
r ite tur 1 virtu 1 worl . e Mitologies proje t i n ttempt to utilize t i 
te nolo y me n o rti ti expre ion n to explore i tori 1 politi 1 

mu i 1 n vi u 1 n rr tive . Mitologies r w in pir tion rom Ire pool 

o liter ry n rti ti our e y pturin t eir intertwinin rel tion ip in 

inem ti orm en e m kin onne tion to t e tron n rr tive tr ition 

o ot er me i u film n liter ture. Mitologies i t e ulmin tion o n 

exten ive o y o work ot n rt proje t n o tw re e i n prototype 

in virtu 1 re lity. 

e t em ti ontent o Mitologies r w in pir tion rom me iev 1 n on to 
ontempor ry liter ry en e vor . e work i loo ely e on t e ret n myt 

o t e inot ur t e Apocalypse or evel tion o t. o n nte Inferno 

urer woo ut ter t e po lyp e n or e Library of Babel. u i 

rom ner Der Ring Des Nibelungen i u e moti to tru ture t e 

n rr tive. e work explore t e eni m ti rel tion ip etween tee our e 
n pture t em in mi e-en- ene t t i roote in t e illu ioni ti n rr tive 
tr ition o ot er me i . 

e i n prototype in virtu 1 re lity Mitologies i n rtwork re te 
or t e (tm multi-per on room- ize virtu 1 re lity y tern evelope 

t t e le troni i u liz tion L or tory ( L o t e niver ity o llinoi 

t i o 5 . e i ten y ten y ten oot room on tru te o 

t ree tr n In ent w 11 . r k nyx wit two nfinite e lity n ine rive 
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3 ri oussos n ish m izri 

t e i re olution tereo opi im e w i re re r-proje te onto t e w 11 

n ront-proje te onto t e floor. Li t-wei t L tereo 1 e re worn to 

me i te t e tereo opi im ery. tt e to t e 1 e i lo tion en or. 

t e viewer move wit in t e onflne o t e t e orre t per pe tive 

n tereo proje tion o t e environment re up te . i pre ent t e u er 

wit t e illu ion o w Ikin roun or t rou virtu 1 o je t . our pe ker 

mounte t t e top orner o t e provi e u io. e u er inter t wit 

t e environment u in t e 3 w n imple tr ke input evi e ont inin 
joy ti k n t ree utton . e w nt en le n vi tion roun t e virtu 1 
worl n m nipul tion o virtu 1 o je t wit in t t worl . Mitologies 1 o 

run on t e m Her more port le ou in t e mmer e k (tm n 

mmer e k2 (tm . 




Fig. 1. rti ip nt inter t wit Mitologies on n mmer e k (tm virtu 1 
re lity y tern. 



e r teri ti o t e ove te nolo y were t ken into on i er tion 
w en e i in to evelop Mitologies . e provi e n ppropri te virtu 1 

re lity pi t orm m inly e u e o t e non-intru ive n ture o it r w re n 
t e ility to provi e roup experien e. i i o re t import n e to i it 1 
work o rt emp i n e iven on t e work wit out worry o t e te nolo y 

overpowerin it . 
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2 Description of the Virtual Experience 

e wor Mitologies erive rom t e reek wor mito t e t re ri ne 
r nte e eu to elp im fin i w y out o t e ret n 1 yrint . e 
viewer in Mitologies re-experien e lie ori lly t e journey o e eu ut 
loo not er i tori 1 n liter ry fi ure nte li ieri 2 . e n rr tive 
i intro u e in torytellin ion tru ture 1 r ely unknown to virtu 1 
re lity worl ut mili r in ot er me i u film n liter ture. 

t e n rr tive pro ee t e inte r tion/inter tion o Ion n en le 
n vi tion lurre llu in tory In pe n o y p ew y eler te 
n e eler te motion u en it rom olor to 1 k n w ite eerie 
neri n mu i iz rre n e torte e or 1 yrint ine tru ture n p r oli 

tyle tempor 1 pre ure n p ti 1 i ontinuitie en e t e viewer in kin 
o tion plot r m tize t e journey n u e t e over 11 n rr tive wit 
re m-like eel. 

e u ien e enterin t e i initi lly lo te on t e nk o river 

in rk ore t n llu ion to nte in ul ore t. 




Fig. 2. e openin o t e n rr tive on o t tr vellin t rou rk ore t 

i remini ent o nte Inferno. 

rom i t n e t e viewer e r t e re kin oun o woo en o t 
n t e u tie oun o w ter w in in 1 1 e river nk . e o t lowly 
ppe r le y t tue mo el o on tello u one 9 . n t i e ir il 

omp nyin nte into Inferno! t e o t ppro e t e ore t e viewer 
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i wi tly tr n porte onto it n t e journey own t e river e in . n t e 

pyi Ip eote two en e re pi e in onfi nr tion t t 

orre pon to t e virtu letote ot. enete illu ion o tr velin in 
t e o t i re li ti t e viewer look own t t e floor o t e n ee 

t e virtu 1 o t w yin ene t t eir eet. 

e intention o t i openin ene i to e t li n expli it en e o tory 

line n rr tive. e low n moot flow o t i intro u tory equen e i let r- 

i n me it tive ettin t e p e t e work eek to ompli t rou out. 
e openin river moti rom i r ner oper ein ol u e or 

t e river ene ontri ute to t e impen in en e o n er n i ten t e 

expe t tion o t e unknown yet to ome. pon lo er ex min tion o t e vi u 1 

n u itory met p or t e p rti ip nt my e in to re o nize t e element 

n t rt r win t e onne tion t t will ui e t eir explor tion 1 ter on. 

e river ene eventu lly e out w ile t e next ene t t o ri ter 
n more et ere 1 p e e in. e tr n ition i i e y oun n ex erpt 
rom i r ner openin e ment o ein ol w i erve 

tru tur 1 moti or t e un ol in n rr tive. neinti p ete viewer 
i em rke rom t e o t n n now t rt u in t e inter e evi ete 
3 w n to ontinue t e journey y n vi tin t rou 1 r e pi ne. r in 
t e i t n e m nifl ent ur urroun e y orti ultur 1 m ze r en 
ppe r . 

ern terliin pire y t e even ur e e ri e in t e po - 
lyp e p e u myrn er mum y tir r i il elp i n L o i e 
n Mitologies 11 even o t e e ur e re repre ente t i one r n ur 

w i i mo ele ter Leon r o in i ket o ur t t w never 
uilt 4 . ny ttempt were m e in t e en i n e to uil in i ur 

ut t e e ttempt never m teri lize . t i or t e fir t time t t in i 
ein n fin lly t ke pe 3 tru ture w ere t e viewer t e op- 
portunity to tr vel up i n roun t e intri tely et ile ri k ome n 
ex mine in i m nifl ent r ite tur 1 vi ion. 

t e oor o t e ur lowly open t e interior reve 1 t e el or te 

p e o ompletely i erent tyle o ur e t e re t o que o or o 

in p in 6 . 

e oun moti rom ner in inten ifle t e en e o elev tion n 

pro re ion rom one p e to t e ot er. e viewer t e ree om to tr vel 

in i e t e re li ti repre ent tion o t e mo que t low level to Imo t eel 
t e rpet or fly i up over it re. wit t e ur t e textile n 
orn ment 1 et il o t e interior o t e mo que re r wn rom v riety o 
our e rel te to t e mo el t e perio n t e t em ti ontent ut 1 o 

pte to t e un ol in o t e n rr tive. e pro re ion i inten ifle urt er 
w en t e rker n more orn ment Ip eote mo que e ome t e entr n e 
to t e even rker n my teriou 1 yrint un erne t . 

n e et morp o e vi note t t e lu uilt ou e in w i 
eonueteuulp e n e eive t e eye wit onfli tin m ze o 

V riou w n erin p t 1 . n Mitologies myri tr n e rk n mi le in 
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Fig. 3. orti ultur 1 r en m ze urroun tern nifi ent ur w i i 
mo ele ter Leon r o in i ket n n e experien e in Mitologies. 

p e re on tru te to re te 1 yrint remini ent o t e 1 yrint uilt 
y e In . el yrint i we or r izome every p t i onne te wit 

every ot er one. t no enter no perip ery n no exit e u e it i 
potenti 1 infinite 10 12 . t e viewer pro ee t rou t e m ze t ey fin 
t em elve on p t t t le to me iev 1 urio ity room room e on 

urer woo ut o t e po lyp e 7 n room popul te y t tue n 

i on room t t require t t t e viewer m ke oi e in or er to pro ee . n 
ome e to pro ee rom one p e to not er t e viewer mu t m ke t e 
ri t oi e . e fir t room or in t n e pre ent t e viewer wit t ree wor 
nte e eu n ri t. t e letter in ny o t e wor i o en 

t en t e oor le in to t e fir t urer woo ut i opene . t erwi e one o 

t e ot er oor le in to urt er p e i opene . t ny point n epen in 

on t e oi e t e p rti ip nt my experien e eit er woo ut or one o t e 
ot er pe i 1 room . 

o t e woo ut room rin to li e in 3 one o even po lypti 

woo ut y urer t t were ele te or t i work. e fir t room en ountere 

i t e even rumpet woo ut room, t pre ent t e ook wit t e even e 1 

r pi ly un ol in on roll w ile t e lou trumpet oun rom ner 

ein ol re juxt po e ; in t e po lypti om n room em le voi e rom 

ie Ikurie ollow worn n tor o w ter t rt to ri e eventu lly fioo in 

t e room; t e our or emen room tr n 1 te t e orror o t e mo t mou 
n ever popul r eet o urer po lyp e wit t e violent motion o t e 

or emen t e our- olore w 11 lo e in on t e viewer; in t e penin o t e 

5t fit e 1 room multiple emi-tr n p rent 1 yer illu tr tin t e lower p rt 

0 t e woo ut ri e rom t e roun like 1 e ; t e orture o t. o n t e 

mo t unu u 1 o urer woo ut t t i not p rt o t e po lyp e n rr tive 

1 re lize wit mo ern interpret tion o torture in t e our ro -like p e 
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o t e room; fin lly t e woo ut room o t. i el i tin t e 
otel tinte erie o woo ut room i pre ente wit wor 
ook o niel _N _N _K L _ _ N. 

yll le _N me n o num ere t y kin om n fini 

next _K L _ _ N me n ou rt wei e in t e 



r on one 
rom t e 
e fir t two 
e it; t e 
1 n e . 




Fig. 4. e t ree- imen ion 1 mo el o t e mo que o or o in p in 
experien e in Mitologies. 



t er room in t e 1 yrint ttempt to pture t e my tery n e uty 
o t e popul r me iev 1 urio ity room . e et p y i / tronomy room in- 
vite t e viewer to ze t t e i tine pel p intin o ell t rou t e 

eyepie e o Ire tele ope; in t e u i room t e viewer n pi y one o 

our in trument n row e t rou t e ore eet . enet eorpy 

n 1 emy room re em le mp tu y room w ere knowle e i 1 ifie 

t rou el or te t xonomie . ntefirtotee room t le o in e t re 

pte rom me iev 1 entomolo y ook . e eo r p y room i pi y t e 
e uty n ur y o me iev 1 rto r p y t rou t e numerou ex mple 

o m p in lu in entr 1 terre tri 1 lo e mo el repre entin t e un men- 

t 1 o tolemy eo r p i 1 y tern. e 1 tter room t e room o 1 emy i 
tur te wit t e ten wor o o mentione in t e 1 . 

o t e room in t e 1 yrint involve re ul re e r on ernin t e 
rti ti ontent well t e i tori 1 n politi 1 ontext t ey repre ent. 
e virtu 1 implement tion owever oe not ttempt to per e tly re re te 
interpret or re lize t e ontext o t e e room ut to pture t eir emotion 1 

e en e. i pro e i e t illu tr te t rou n ex mple on one o t e room 
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o t e 1 yrint t e o room ter ieronymou o . i room rin 

to li e o mo t mou work e r en o rt ly eli t (1505-1510 . 

i tripty ow tern ter t i e t. e entr 1 p nel w i i t e 

ujetote pew rm wit t e r il nu e li ure o men n women 
portin li entiou ly in p nor mi 1 n pe t t i tu e wit nt ti 
rowt o qu i exu 1 orm. o eem to ow eroti tempt tion n 

en u 1 r tifi tion univer 1 i ter n t e um n r e on equen e 
o ori in 1 in u um in to it n tur lly e i po ition. e u je t re 

erive in p rt rom t ree m jor our e e iev 1 e ti rie lemi prover 
n t e t en very popul r re m n o ult ook 11 mixe in t e meltin pot 

o o toun in ly inventive im in tion. 

e ve o en to re lize t i entr 1 p nel or t e virtu Ip e. ym ol re 
ttere plenti ully t rou out t i p nel. ne o t e mo t in tin ym ol 
it to ouple in 1 lo e w i illu tr te t e prover oo ortune 
like 1 i e ily roken. el lo e i re re te in t ree imen ion n 

re re te 1 o in t ree imen ion 11 loewtte tripty w ole 

repre ent . irtte lepr ieote worl etween en n ell. e on 
tee ret o 1 emy n it lie ori 1 me nin . e viewer i le to enter 

tel lo e w i repro u e t e movement o t e e venly o ie . t e 
viewer n vi te rom one lo e to t e next t e w 11 r u lly move kw r 
to reve 1 multiple lo e ri in like ir u le . r t er et ere 1 r ment rom 
ner oper otter mmerun i u e to illu tr te t e over 11 lie ory o 
t e p intin . 




Fig. 5. view o t e o room one o t e more t n 30 room in lu e in 
t e Mitologies worl . 
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e on tru tion o t e room in In e omplex we o met p or n 
in. e p t rom one room to t e ot er m y e line r ir ul r or truly 
1 yrint ine epen in on t e viewer oi e . en t e 1 t room o t e 
1 yrint i fin lly re e t e viewer te ion journey on In e . e pe 

0 t i room re em le t e num er ix or t e ell o n il o t e two 

room t t pro ee it. e viewer enter t t e n rrow tip o t e room n ir- 

le roun it until t e enter i re e w ere t e en ounter wit t e minot nr 

im el t e ym ol o e t t ke pi e. e repre ent tion o t e minot nr 

e te in i m nifi ent temple i e on e re ip e t met p or rom 
onolo i 11 . e me te w it 11; we ten tow r ommon o 1 

Ik e t w o 1 im 11 un er i power. 

e per onifi tion o e t keleton lie on ier wit in n el or te 

t Ique e or te wit kull n m ny 1 mp . e i wr ppe in ri ro e 

( e t ke w y t e ri m n we It n we r 1 urel rown ym olizin 

i rule over 11 mort 1 . n one n e ol wor entwine wit n olive 

r n me nin t t pe e nnot en ure i men o not run t e ri k o e t 

in fi tin or it. e motto ove t e t Ique re e t m ke 11 men 

equ 1. our li te t per t n on eit er i e o t e ier. Not r nk nor i nity 

n me wit t n ; y power exten o er every In. 11 

n t e lower ore roun t n two putti e vily veile to i ni y t eir lin - 

ne not t eir mournin or t ey repre ent m n pie ure in t e t in o t i 

worl . e power n lory o t e worl re own in t e il we rin 1 i 1 

rmor n rryin rown mitre epter n me 1 on u ion w ile 

wor Ine n m r 1 ton lie t i eet. e ot er putto repre- 

ent 11 um n invention n rt; t e fl il n ovel lyin ne r im t n or 

ri ulture w ile t e quill pen roll p lette n ru e n omp n 

tri n le t n or t e rt n ien e . n one i e o t e room p ir o w ite 

e t ere win uilt y e lu 1 y on t e floor. 

ile t e viewer will urely ttempt to ppro t e minot ur i en 

rypt open un er t eir eet en lin t em to e pe rom tel yrint y 

u in t e p ir o win . n e in wit tee innin equen e n vi tion 

1 i le n t e viewer i r e own into ti 1 pool w ile t e oun 

ex erpt rom ein ol re e lim x. nme e in m enin pir 1 

w ere voi e n oun wirl own Ion wit t em t e virtu 1 re lity viewer 

ve fin lly 1 n e k on t e o t on w i t e journey t rte out rom. nly 

now on tello t tue i no Ion er t n in on t e ow o t e o t. n te 

ew ttere e t er remn nt o t e win t t le to t e e pe ve repl e 
t e monument 1 fi ure. e voi e n oun moti ve 1 o Ite wit only 
t e re kin oun o t e o t re kin t e ilen e. e n rr tive t to 
1 yrint ut 1 o 1 yrint in it el omplete it y le. 

3 On the Use of Narrative and Interactivity 

Mitologies t ke r i lly i erent ppro to n rr tive tru ture n one 
oul y Imo t i nore inter tivity. n ert in e t e u ien e no on- 
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trol u in t e i tin t openin w ere t ey re t ken or t e o t w i 
lowly tr n port t em to t e t e r 1. rom t i point on one p rti ip nt m y 

t ke ontrol over n vi tion w ile t e ot er experien e t e virtu 1 n rr tive le 

y t i per on oi e . e inem ti n rr tive orm pre erve it el t rou 
t e ontinuou low pen pro re ion ieve rom one ene to t e ot er. 

e 1 yrint pre ent it vi itor wit oi e yet 11 oi e re in e en e 
illu ory t ey ultim tely le to t e me fin 1 on ront tion wit t e mino- 

t ur t e 11 t rou t e tr p oor n t e return k onto t e o t t u 

ompletin ir ul r journey. 




Fig. 6. e eo r p y room fille wit me iev 1 m p o unique e uty n 
ur y in lu in t e Planisphaerium sive universi totius ele ti 1 rt 
illu tr tin t e un- entre y tern outline y operni u . 



nter tivity i r i on etre o virtu 1 re lity worl . o t people ow- 
ever o not know or un er t n ow to e 1 wit inter tive omputer- e 
rt let lone wit inter tivity in immer ive n in m ny e omplex virtu 1 

worl . e virtu 1 experien e i i orientin unn tur 1 n i ult or mo t 
even i t e te nolo y u e i imple n n tur 1 it urrently et . e 
le troni i u liz tion L or tory een tively involve in efinin t e 
uture ire tion o virtu 1 re lity te nolo y t rou t e evelopment o t e 
w i one n r ue ein one o t e etter ex mple in t e mut o 
inter tive virtu 1 re lity y tern . ur p rti ip tion in multiple venue pro- 
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vi e u wit numerou in t n e to o erve t e re tion o people inter tin 
in t e environment w et er il ren ult in le viewer roup ex- 
pert or novi e viewer . e e o erv tion le u to elieve t t n inter tive 

experien e i not ne e rily 11 1 t m tter w en re tin work o rt unle 

t e work it el i out inter tivity. y woul we ne e rily nee re tive 

omputer pro r m in or er to m ke n rti ti t tement ri in met p or? or 

u w t i more import nt i t t t e viewer will not e on ume y le rnin 
ow to inter t wit t e te nolo i 1 inter e. urt ermore it i import nt or 
t e rti t t t t e viewer will urp t e t e o in tion wit t e me ium 

w i m y i tr t rom t e ontent o t e work it el . ert inly no p inter 

woul on i er er work u e ul i in te o t e work t e viewer e me 

in te wit t e on tru tion et il o t e nv . t i t u riti 1 or 

virtu 1 re lity to move eyon t e level o t e te nolo y. n t e mi ion o 

work o rt i to pre ent lien e to t e prev ilin ort o oxy o virtu 1 

worl . 

ow o we re pon to t i mi ion o i tin t e orm n rti ti pro e 
o virtu 1 worl re tion w y rom t e te nolo y n into t e ontent? t i 
not t e intention o t i work to re li ti lly re re te r ite tur 1 worl o 

t e p t nor to oil p e time perio into n ttempt to re efine t em p rt 

o on u in ort n r mente experien e. It ou t e pie e involve 
re e r in i tory myt olo y liter ture rt n mu i it voi t kin 

i ti or en y lope i t n e. e intention i to re te work roote in t e 

um nitie w i trie to urp t e u u 1 orm o virtu 1 re lity worl t e 

orm o n vi tion t rou tr t eometri pe . not er le r intention o 
Mitologies i t e ttempt to impo e n rr tive ontent n tru ture. e film- 

like mi e-en- ene w ele te ot or it mili rity wit t e viewer n 

mo e o expre ion. in lly t e oil or tion etween in ivi u 1 o i erent 
intere t n i ipline to mel to et er into work t t m ke o erent 
w ole voi in w t i requently t e re ult o oil or tion or t e re tion o 
virtu 1 worl erie o i erent non- o i te worl onne te y port 1 . 

4 Conclusion 

po lypti liter ture n rt ve Iw y een on erne wit t e ppro 

o t e en o t e millenium n o oe ientifi n popul r liter ture out 

virtu 1 re lity. t i work i re te tow r t e en o t e twentiet entury 
it inevit ly inve ti te t e me nin o t e Apocalypsis in u ontext. or 
Mitologies virtu 1 re lity i u e ve i le to liter lly expli te t e exe e i o 

t e evel tion. n t e Apocalypse ou le per pe tive pi y n import nt p rt 
in uil in n exp n in t e met p or . n one level t e vi ion i everely 
re tri te n r mente n u er on u ion. n p r llel level it i le r 

n on i e. e met p or re t on e in le n ou le; t ey in orpor te 

1 rity n on u ion unity n multipli ity rti try n o . n i tive i 
t e element o t e 1 yrint ym ol o t e tom o e t o ell ut 1 o 
pi e o ju ment n wor ip. 
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owever omplete n exten ive Mitologies i till inten e n experiment 
wit t e t te-o -t e- rt in te nolo y ut 1 o n ttempt to ee t e twen- 
tiet entury worl rom per pe tive ot er t n t e te nolo y it een 

re te wit . It on it i le t up to t e viewer to ynt e ize t e i tori 1 

liter ry or myt olo i 1 m teri 1 into ontempor ry relev n e it i till t e o- 
u o t i work to re te n Item tive to t e u u 1 t-p e oi e- riven 
ynt eti worl n m ke pro oun n 1 tin imp t wit in in pe ifi 
n te nolo i lly- e rt in ener 1. en Mitologies my e on i ere 
u e ul experiment in rt. 
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Aggregate Worlds: 
Virtual Architecture Aftermath 



Vladimir Muzhesky 



Abstract. In the following article the phenomenon of virtual world is analyzed 
not as a point of reference or relation to existent meaning deployed elsewhere 
hut as a point of semiotic and perceptual construction of synthetic content allo- 
cated for use via computer network. Author extends this analysis into the legal, 
economic, and neurological discursive lines of interaction, monitoring and con- 
trol. Aforementioned aspects inherited by contemporary virtual worlds from its 
military predecessors in many cases were used as landmark metaphors by de- 
signers to refer and/or reframe existing linear discourses by means of interfac- 
ing them in a non-linear way from within virtual and augmented reality envi- 
ronments. This article is concluded with a scheme or project for an infrastruc- 
ture of aggregate world, as it appears to be functional and formative in its rela- 
tionship with interactive content. 



Perceptual Synthesis, Content, and Aggregate Locality 

Digital or cyber-space commonly referred to as virtual or synthetic locality is 
in reality underdefined concept. Epistemological aggregate or defined status of spati- 
ality which is implicit to locality as concept is channeled via semiotic zones of refer- 
ences to real or memorized as real situations in virtual environment. Hence, to define 
virtual space we have to follow the pattern of interference emerging from mediated 
life streams of real worlds and rendered objects and its interrelated neurological, per- 
ceptual, semantic, and economic contextual aspects of spatiality. 

From this point of view virtuality is a complex transparent conglomerate: 
current 3D dummy realms with limited interaction and abused liberating tendency can 
be interpreted as desubliminization of mental tangibility inherent to digital space - it 
is thrilling how conceptually loose its constructs can be. It seems they are projected 
into absolute nothingness simultaneously being declared as hi-tech spectacle, sort of 
industry-dimensional test field with default meaning categories like a body shape 
which stands for a biological connector and shooting which represents informational 
transmission. 

Described by Marcuse still back in the 60’ s disubliminization tendency of 
media obviously integrated itself into virtuality. Resulting alienated toy-like industry- 
dimension conceals the aspect of virtual world as a space to which knowledge is 
functionally and structurally bound in the first place. When we analyze this space we 
reconstruct it, we project it and provide it with biologically streamed content. This 
content in its turn is being deployed when we “act” in digital space in accordance with 
electronic and perceptual semiotics which might have become a starting point for 
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digital economic theories would we have left astride Freudian limits of toy-humans 
(avatars) in virtuality with all their body based pseudo-topologies (virtual embodi- 
ment theories) and include pattern recognition and recombination as a common sphere 
of action when it concerns biological and electronic interference. Of course we can 
stream raw biological situation as any other kind of raw data material, but how long 
are we going to follow graphic or academic industry preset attitude to spatiality and 
impose its mental-proof rendered polygons onto ongoing perceptual-electronic de- 
ployment? 

Semiotic mechanism of streamed and rendered synthethis together with its 
perceptual properties constitute specifically synthetic meaning and defined aggregate 
locality. To follow discursive line of aggregate construction of digital space as an 
alocation of mediated content means to encounter spatial paradox represented as 
screen border. With the development of interactive media screen border seems to 
position itself as a materialized limit of content; almost Cartesian zone of dual visibil- 
ity, virtual perspective of inside and outside of the machine content performance 
being simultaneously exposed to the user. In the latest artificial intelligence reviews 
one can find the reference to this zone of content visibility. 

Take for example Douglas Hofstadter analysis of Stanislaw Ulams opinion 
about perception as a key to intelligence. Ulam wrote: “you see an object as a key. ..it 
is a word as which has to be mathematically formalized. Until you do it you will not 
get very far with your artificial intelligence problem.” Hofstadter concludes his re- 
search in the following manner: “in any case when 1 look at Ulams key word 
"as" I see it as am acronym for “abstract seeing.” 

The analysis of limits and constructs of an aggregate (locally content- 
defined) virtual world in terms of “abstract seeing” interface of interactive content 
displays another controversial feature of synthetic reality: content represented or con- 
structed within a virtual world simulates or refers in its simulation to the structure of 
physical action leaving perception out of discursively preprogrammed visibility. This 
sort of ego-like phenomenon of virtuality is usually addressed as “virtual conscious- 
ness.” However what it really comes down to is probably an early process stage of 
alocation of synthetic meaning which as a restricted (in Bataille’s sense) economy 
follows Marxist-Hegelian dialectics of selfrefletive phenomenological emergence. In 
this frame, virtuality is constantly defining itself via neglecting the transparency of its 
production and reproduction when it comes to its origins in real or memorized as real 
situations. Current transmission from virtual (rendered-only) to augmented (streamed 
and rendered) reality in computer science (augmented reality and active telepresence 
system research) reflects crystallization of aggregate phase in virtuality. 

It seems that there is an interface of perception missing as an alternative to 
the existing and widely abused interface of action in digital culture. This gap in digital 
economy makes us look for in depth scaled content in simulation as synthesized per- 
ception as opposite to the surface interactivity as a synthesized action. 

In neurocognitive context it is possible to analyze the synthetic interaction of 
human and electronic system, with the following extension of the analysis into con- 
ceptual, economical and political faculties. And to the same extent as neural research 
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influenced social sciences, the models of neural networks bringing out a necessity to 
define fhe infrastrucfural elemenfs of human infelligence in ferms of its vision and 
activity, in digital culture neural network and related representational models play 
crucial role when it concerns virtual content. 

From Heidegger to Baudrillard there was a strong tendency in philosophy to 
highlight the influence of technologically mediated and even economically and ideo- 
logically simulated (in case of Derderrian's analysis of espionage structures) social 
terrains. From this point of view, current mental alienation of the role of perception 
and recognition in virtual architecture, is even more appealing, than physical labor 
alienation of the dawn of fechnology. Historically worker-machine dichotomy 
brought the achiasm of society and industry as a point of theoretical reference. To- 
day’s tendency to unify fhis dichotomy into a cyborg bioelectronic complex concealed 
them. The tension of firsf lapses of indusfrial maps and dimensions was resolved 
wifh geopolitical pattern recombination of industrial and socio-industrial revolutions, 
while cyborg- situation as the closure of revolutionary localities should be resolved on 
the metalevel: the selfdominant for synthetic concepts deployed by means of virtual 
forms, links and inter-forms should be defined and neurolized, embedded in users 
processing with existing one dimensional ideologies, economies and politics being 
referred and mentally attracted to this new field. 

The fact that neuralization substitutes the revolution in the context of virtu- 
alized social sphere, speaks for if self in ferms of spatial properties: vital social activi- 
ties such as banking (online transactions), education (online courses and virtual uni- 
versity projects), research (for example, Sfanford KSL project which provides access 
to running research and development software systems over the World Wide Web) 
are pressed out into the virtual dimension. New plane of immanence is nof a frozen 
pattern of biosocial oscillation, with which philosophy can easily play, it is a dissipa- 
tive and in some cases selforganizing ferrain of attractors, which implies fhaf af least a 
part of its organizational points are useless. In terms of neural networks, new emer- 
gent attractors of pattern recognition activity are referred to as spurious memories. 
Net as a bioelectronic phenomenon has plenty of fhem. If is wilderness for Freudian 
experiments on wired population, and at the same time an avatar of escape from re- 
currenf dissublimizafion. Projected onto the bioelectronic plane, dissublimization of 
the net spurious memories connected with the exposure of percepfual intensities in the 
logic of virfual worlds brings up a new architectural form which inferfaces confent on 
a subversively deeper political level fhan any ofher existing form of synthetic activity 
as much as it embodies synthesis of content itself by means of percepfual interfer- 
ences and modulations. 



Hyperreality Neurolized 

Wifh the discovery of neural systems, the ability of single class nefworks to 
generate additional or spurious memories was classified as generalization activity, 
when involved in the pre-recognition of new stimuli (Ezhov, VVedensky, 1996). This 
aspecf of neural nefworks corresponds to the immune complex of human systems. 
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where antibodies are preliminary generated in order to establish a binding with a 
potentially new antigen. 

Virtual complexes and communities, those neo-homo-ludus kind of things, 
and also those which are designed to "reflect" the actual structure of social architec- 
ture and in practice, pre-recognize social tendencies and statements (this is actually 
the essence of any simulation) are the best indicators of spurious memories grown on 
artificial terrain of virtual reality. They remind an interpretation of chimerical patterns 
in human dreams, when imaginary cities are interpreted as real. For example, imagi- 
nary city of London is taken as if it were London. If one considers new attractor- 
spurious memory effect, the phenomenon of "as if London" becomes possible only, if 
the informational system, revealing its resemblance with immune complex has gener- 
ated the class "London" in advance. The same concerns virtual architecture on the net, 
which does not of course reflect, but replicates the class of Amsterdam or London in 
virtual reality databases. Human operators have to assume that the city, which they 
build is Digital Amsterdam before the spurious representation is architecturally real- 
ized and installed on a server. 

Consequently, virtual architecture takes over a replication-transformation 
function of the neuroactive compounds in the process of reintroducing an agent of the 
outer environment inside neural networks, and second, of the revolutions by reorgan- 
izing knowledge-production bound communities on the virtual basis. 



Nodality as Architectural Framework 

The concept of node can be regarded as a focal point of interactive content 
deployment in aggregate world. Nodality as a synthetic spatiality is a tangible multi- 
level topological framework, which relates to the concept of node as location interfer- 
ence within the context of informational technologies and addresses its architectural, 
perceptual, and conceptual properties. Based on the internet as a temporary zone of 
preprogrammed interference node marks a point where traditional monospatial semi- 
otics is confronted with transgressive multispatial discourses of aggregate architec- 
ture. 



Interference 

Classes of virtual spatial language are organized by location in both human 
and artificial information networks, and not by qualities or properties. To the same 
extent as neurointeractive compounds reintroduced outer environment in perceptually 
illegal way (avoiding the filtering of the outer senses) in-forming human system, the 
simulation reintroduces it in conceptually alegal way and reveals that, as much as 
spurious memories are utilized, aggregate spatiality is first of all a neuroactive com- 
plex. As such, like many other neuroactive natural or artificial compounds, aggregate 
architecture possesses structural key to the neural informational processing. This 
phenomenon is not and cannot be controlled by either human or artificial intelligence, 
it is extremely autonomous, to the extent of aforementioned conceptual illegality: 
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from the point of view of the law the act, for example, of currency analysis (to use 
recent research from Berkeley Experimental Laboratories where a legal situation itself 
became a subject to augmented reframing by means of substituting the agent of action 
with a user-robot interface) destruction, and who knows, may be replication on the 
internet can be interpreted as the act of crime or of art, depending on the extent of 
reality. However, this is exactly the point, where semiotically based intelligence of the 
law is decomposing itself under the influence of higher informational structure: vir- 
tual index of membership and action leaves no unified space, time, and agent of ac- 
tion. This is were augmented reality research and robotic manipulation involve indus- 
trial dimension of virtuality into following the same remote lines of vision and power, 
interfacing data in a non-linear way and thus destroying linear discourses of legal and 
economic laws and presupposed unified conceptual entities. 

Once the factor of perception and interfaced content enters the analysis the 
laws of unified subject of action become ambiguous. It would not be an exaggeration 
to say that hyperreality as a phenomenon of neural network simulation is censoring 
every possible plane except of spatial-information terrain. In this context digital cul- 
ture relates to speed through the deconstruction of the carrier (similar to the futurism 
movement and scientific art at the beginning of the century and cold war carrier de- 
struction paradigm of missile engineering in the 80’s-90’s) as the most abstract repre- 
sentation of system transformation. 



Speed 



There is a tradition in post modern philosophy, which regards speed as a 
formatting factor of economical hyperterrain. This theory was thoroughly developed 
for many areas, however it is appropriate to focus here primarily on relevant to virtu- 
ality applications. Among them there is philosophy of geography (Olson), the phi- 
losophy of espionage (Derrderian), and the theory of general economy (which did not 
make it yet in to the textbooks according to CAE) developed by Bataille. Erom the 
analysis of the structure of espionage (Derrderian) and development based world 
maps (Olson) it was derived that speed can be understood as a hyperfactor in reflect- 
ing and reorganizing reality. Extending this conclusion into the electronically medi- 
ated cultural sphere: we can investigate what role the factor of speed plays in artificial 
environment; and how it influences interactivity as a cultural relation to image. 

By definition, synthetic or virtual nature of any aggregate simulation makes 
it structurally dependent on its own speed. In other words, in order to differentiate 
between relative positions within virtual reality and integrate them into a unified rep- 
resentation such factor as information velocity ascribed to a certain locality has to be 
considered in the first place. In other words there is a major semiotic difference be- 
tween positioned and represented meaning within the logic the logic of virtual words, 
the latter being exposed to limits of its speed momentum which it is programmed to 
acquire in a given space time continuum. 

Within above described framework, the concept of interactivity acquires an 
interesting property: it changes a polarity on the scale of editing. In general, interac- 
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tivity can be understood as a socially, culturally, and informationally preprogrammed 
approach to eidos, which defines correspondent modalities of representation. How- 
ever we can delineate a hypothetical architectural research where modality of speed is 
defined before interactivity, by means of ascribing different informational velocity to 
different nodes of a virtual world. Thus, aggregate architecture can become a content 
formatting factor, which induces a cultural inversion of the relation to image. 

There is another edge, in this approach, which relates to Bataille's cultural 
theory. Decades ago French theorist described a theoretical model of the economy of 
waste as a counterthesis to existing restricted economy. Applied to the context of 
synthetic environment and speed as its formatting property, Bataille's theory finds 
more stable grounds, not only because it is a digested noematic simulacrum reality, 
but primarily because the topography of speed in its relation to information velocity 
provides perceptual material which is essentially different from that of raw material 
reality even when it is being streamed, and which can support unstable under ordinary 
conditions cognitive constructs. One of those is an idea from the dawn of AI studies, 
which describes the network of users working with shared databases as a global tech- 
nocerebrum. It is culturally outdated, but not yet if one is (to use Timothy Leary's 
term) disattached within shared virtual reality environment and plugged in to Indus 
like multiple crainial gear. 



Alegality 

When we face the situation, where points of stability of legal reality, such as 
a subject of action with its unified complex of personality, spatio-temporal locality 
and linear logic are not transmitted through the nodality of Aggregate synthetic envi- 
ronment, we have to question the causation of this alegal resistance. Imagine an ille- 
gal action realized via remote control machinery interface, which the net essentially 
appears to be; what would be the basis to determine whether the subject of crime was 
only one person, or a whole group, where each member was responsible for a certain 
algorithmically step of illegal operation. Furthermore, the personality of the virtual 
criminal, and hence motifs of crime, also remain quite vague, especially, if we con- 
sider that all representations including agent of crime, instruments, and actions can be 
programmed in any space-time, hence numerous replicants of these representations 
are as much real on the net, as the ones which actually disturbed the law. Finally, in 
between when and were the complex of computer commands was launched and the 
period and place when and where it was executed there can be millions of miles and 
hours. The last kick in the shorts, which the remotely manipulated metal boot of the 
net gives to the legal system, is that all this may be a self-organized process, triggered 
without direct human interference. 

In this respect the phenomenon of virtually interfaced content generates a 
membrane effect, which just doesn't let the legal system through, partially because the 
latter appears to be based on the strictly human body spatiality, which implies physi- 
cal disposition of action and meaning. But one important thing is denied by the legal 
discursive dimension: the sphere of vision, which as an essential part of monitoring 
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has to be pushed away from legal interrogation, because otherwise it would interro- 
gate itself creating the infinite spiral of metareflections. 



Perspectives 

The representation of the legal one level reflection logic of mass produced 
virtual reality is one way membrane of the screen: creating the illusion of depth the 
screen seems to counterfeit the conceptual plane of phenomenological philosophy. 
Being the base of perceptual vision on one side and henceforth being embodied in 
human perceptron, the screen is constrained of vision on the other side, where it is 
embodied in the electronic configuration of technological space. As a hybrid of two 
bodies and two spaces, the screen is exposed to both visibility and invisibility, the 
topic of the last writings of Merleau-Ponty right before his death in 1961. In this era 
of television revival, he writes (his notes were actually entitled "On Visibility and 
Invisibility"): “This is what Husserl brought frankly into the open when he said that 
every transcendental reduction is also an eidetic reduction, that is: every effort to 
comprehend the spectacle of the world from within and from the sources demands 
that we detach ourselves from the effective unfolding of our perceptions and from our 
perception of the world, that we cease being one with the concrete flux of our life in 
order to retrace the total bearing and principal articulations of the world upon which it 
opens.” 



Merleau-Ponty delineates an economy of reflection, which avoids effective- 
ness of acting in the world and connects it with the eidetic reduction described by 
Husserl and reconstructed on the basis of computer simulation. 

Before the Soviet Union collapsed billions of rubles where spent on so-called 
psychotronic research which investigated shared invisibility of control. This is one of 
the factors which critics confusing archeology of media and history of calculus usu- 
ally neglect, that computer all in all in a historic perspective of interactivity was not 
the only available instrument. It is a perceptual emulator of analog instruments devel- 
oped for the variety of purposes. From this point of view a missing constituent for 
synthetic content becomes visible: the absence of neurological feedback in informa- 
tional processing is a physical economic resistance, sort of a digital gravity of a vir- 
tual world. 

In fact, if the net refers to phenomenology, this is only due to its embodiment 
in the mass media, a difficult infancy, so to say. The screen is what it is only because 
the media in its legal form was always constrained of multidimensionality. The 3D 
phenomenon of industry-dimension is as good indication of alegal multidimensional 
arousal, as it is a harmless placebo imposed on our vision by the legal and economic 
system. However, the invisible part of the screen remains uncensored, simply because 
of the fact that legal system can not by definition include an illegal element, although 
it can happen in practice. The question, why the invisible body of the screen is illegal, 
has a simple explanation: because it performs the role of mute representational plane 
of the neuroactive agent, which had been a posteriori censored out of perceptronic 
networks of manipulated cyborg oriented population. 
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The fact that the screen is a perceptronic modulation complex hidden under 
conventional placebo of perspective first received attention in the seventies, when the 
advertisement with the invisible subliminal component was widely introduced via the 
reality streaming networks of cinemas and TV stations. The following prohibition 
measures reflected the pathological fear of the law, when it concerns psychotropic 
effect: even though it was a brilliant marketing technology, subliminal advertisement 
was prohibited, which was against any economic law. What the legal system was 
fighting against was an alternative economic attractor, which may have opened virtu- 
ally a new dimension for the biotechnological interaction, if not only the signs but the 
products were neurolized. 

The evidence from the wide range of neuropsychological studies suggests 
that there is clear dissociation between the visual pathway supporting perception and 
action in the cerebral cortex. In the context of media it implies that the way the econ- 
omy is transduced by the electronic repository of information may imply at least two 
ways, which would accentuate action or perception. The fact that contemporary vir- 
tual media is legally based on action and participates and growth into the implies 
impossibility to transmit the lines of power into the perceptron, as much as it is based 
on different logic than legal linear logic of causative action; thus, the connection be- 
tween monitoring and monitored parts of the population is disintegrated on both sides, 
which gives no basis for conventional electronic reconstruction, but leads to the point 
of social anticyborg biotronic bifurcation. 

Economical disposition of powers in global hiatus is reflected in relations of 
restricted and unrestricted, or horizontal (such as for example Hegel and Marx theo- 
ries) and vertical (such as Bataille's economy of waste) economies. If the former one 
presupposes the effectiveness of action, which a priori can not be fulfilled because of 
the counteraction, the latter one suggests to refer economic constructs to waste and 
thus makes the restricted economy of action mutate into the unrestricted economy of 
vision which establishes basis for economical properties of perceptual products and 
defines virtual reality as a product spatiality and locality. 

In fact, the first thing, which is being violated by alegality of vertical econo- 
mies is one dimensionality of products and production. Analyzed by Marcuse, mono- 
discursive reality by means of repetition binds mesmerized human consumers to the 
single dimension of products, which is not even human, but is referred as human, or 
acts "as if humanified" by the process of mass production. On the contrary the fact 
that a perceptual modulator is embodied into the legal economic routines fractalizes 
the vision of the product to the extent where the borders of real and virtual represen- 
tations vanish among the millions of multiplied resonances of products, representa- 
tions, cogitos, psychological triggers, bioreflexes, etc. 
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Neurospace 

Via the membrane of screen restricted to eideitic representation by means of 
vertical economy we arrive to the neurospace. The economic point of departure, 
which ends up with the rational fusion of neuroactive and economic activities in the 
on the biotronic plane we will call Marxian-Bataillean interface, reflecting its historic 
origins, horizontal and vertical properties, and prospective effects on the society. It is 
a selforganizing loop of visionary economic development, which extends mental 
economic instruments via mass media into the neurospace and organizes mass pro- 
duction in accordance with mediating properties of simulacrum. 

On the basis of its perceptual and economic platform, neurospace can be de- 
fined as an autonomous hypernetwork of inner-outer inferences of informational dis- 
courses. Whether biologically or electronically realized, it theoretically establishes the 
same conglomerate of protomodel space niches leveled by the modes of perceptual 
intensities and, hence, correlated with the extent of perceptronic transformation. 

Further we suggest an architectural demo version of aggregate world struc- 
ture which triggers aforementioned interfacing spatiality. Considering this property 
the model was called Aggregate World. 



Aggregate World: Construction Set for In-Network Users 

Aggregate world is a project or so to say architectural perspective and semi- 
otic showcase of content formatting virtual reality spaces based on the internet. It is a 
research environment into the formative processes of synthetic content as it is de- 
ployed in the spatiality of simulation. 



Location 

aggregator.thing.net 



Structure and Functionality 

Aggregate world functionally replicates human informational processing. It 
is conceptually based on neural network research and its cultural applications in the 
context of informational and social studies. Aggregate world consists of the following 
elements: spurious collectors, transmitters, content traffic, and posted databases. 

Spurious collectors being the first elements which freshly rendered users en- 
counter at AW are single user virtual reality worlds based on perceptually aggregate 
architecture. As opposite to physical buildings, synthetic constructs suggest liquid, 
constantly changing configurations and dispositions of elements linked to external 
events. As such perception and processing of the user becomes a space where actual 
synthetic content is deployed. 
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Following architectural elements of AW are Transmitters which represent 
multiuser domains linked to single user worlds as external events. They provide users 
with possibilities of analog communication based on their previous synthetic percep- 
tual experience. The communication is realized in form of mobile convertible ele- 
ments which can represent either a world (an architectural file where communication 
is placed) or an avatar (a temporary synthetic representation of a user in communica- 
tion) depending on a users intention to host or join communicational situation. 

Posted databases imply publicly accessible documents where users input and 
evaluation is listed. Access to databases is dependent on perceptual input of virtual 
architecture. 



Rendering 

To be rendered by AW means to be perceptually involved in its simulatory 
architecture, channel biosocial representation into synthetic communication, and par- 
ticipate in shared processing and databases. 

AW renders individual and collective digital representations in the same way 
as an image can be rendered on the plane of simulation in content development edi- 
tors. By linking existing representation to a certain moment in abstract visual se- 
quences, AW renders perceptual properties to otherwise invisible discourses of 
power. To a certain extent it is a simulation of Panop ticum, and as such it is a great 
deal connected to the meaningfulness of architecture in its relation to the position of 
the image. Constructed of purely synthetic hyperreal imagery AW distributes mo- 
mentary properties of this imagery in between existing semantic structures ascribing 
new dispositions of synthetic content to borrowed ideological constructs. 



Source and Code: A Few Thoughts 

Aforementioned model may be reinterpreted, used and distributed by virtual 
community builders over the internet. Taking into consideration recent emergence of 
visible programming environments like Visulan it is assumed that any produced code 
is different from the production code. Transparency is probably the only condition of 
aggregate replication. 




Zeuxis vs Reality Engine: 
Digital Realism and Virtual Worlds 
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The story of the competition between Zeuxis and Parrhasios exemplifies the 
concern with illusionism that was to occupy Western art throughout much ofits 
history. According to the story, Zeuxis painted grapes with such skill that birds 
attempted to eat from the painted vine. 

RealityEngine is a high-performance Silicon Graphics computer designed to 
generate real-time interactive photorealistic 3-D graphics. It is used to author 
computer games, to create special effects for feature films andTV, to do computer- 
aided design, and to run scientific visualization models. Last but not least, it is 
routinely employed to power virtual-reality (VR) environments, the latest 
achievement in the West's struggle to out do Zeuxis. 

In terms of the images it can generate, RealityEngine may not be superior to 
Zeuxis, yet it can do other tricks unavailable to him. Eorinstance,it allows the viewer 
to move all around a bunch of virtual grapes, to "touch" them, to lift them on the palm 
of a virtual hand. This ability to interactwith arepresentation may be as important in 
contributing to the overall realityeffectproduced as the images themselves. 

During the twentieth century, as art largely rejected traditional illusionism, the goal 
so important to it before, it lost much of its popular appeal. The production of 
illusionistic representations became the domain ofthe media technologies of mass 
culture-photography, film, and video. Todaythese machines are everywhere being 
replaced by digital illusion generators-computers. 

How is the realism of a synthetic image different from the realism ofoptical media? 
Is digital technology in the process of redefining the standardsof realism determined 
by our experience with photography and film? Do computer games, motion 
simulators, and VR represent a new kind ofrealism, one that relies not only on visual 
illusion but also on themultisensory bodily engagement of the user? Some of my 
previous writings addressed these questions in relation to digital cinema, computer 
animation, and digital photography. In this essay I will discuss a number of 
characteristics that define visual digital realism in virtual worlds. 

By virtual worlds I mean 3-D computer-generated interactive environments 
accessible to one or more users simultaneously. This definition fits a whole range of 
3-D computer environments already inexistence: high-end VR works that feature 
head-mounted displays and photorealistic graphics generated by RealityEngines or 
similarly expensivecomputers; arcade, CD-ROM, and on-line multiplayer computer 
games; low-end "desktop VR" systems such as QuickTime VR movies or VRML 
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worlds, which increasingly populate the World Wide Web; graphical 
chatenvironments available on the Internet and most other major computer networks. 
More examples will be available in the near future; indeed, 3-Denvironments 
represent a growing trend across computer culture, promising to become a new 
standard in human-computer interfaces and in computer networks. 

Realism as Commodity 

The word digit has its roots in Latin, where it means "finger" and thus refersto 
numbers and counting. In a digital representation all dimensions producing the reality 
effect-detail, tone, color, shape, movement-are quantified. As a consequence, the 
reality effect itself can be described by a set of numbers. 

Various dimensions determine the degree of visual realism in a virtual world. Two 
of the most important are spatial and color resolutions-the number of pixels and colors 
being used. For instance, given the same scene at the same scale, an image of 480 x 
640 pixels will contain more detailand therefore will produce a stronger reality effect 
than a 120 x 160 image. Since a virtual world is modeled with 3-D computer graphics, 
the number of geometric points each object is composed of (i.e., its 3-D resolution) 
also has an effect. Once the user begins to interact with a virtual world, navigating 
through it or inspecting the objects in it, other dimensions come into play. One is 
temporal resolution-the number of frames a computer can generate ina second (the 
larger the number, the smoother the resulting motion). Another is the speed of the 
system's response: if the user clicks on animage ofa door to open it or asks a virtual 
character a question, a delay in response breaks the illusion. 

The quantifiability of these dimensions reflects something else: the cost involved. 
More bandwidth, higher resolution, and faster processing result in a stronger reality 
effect-and cost more. The bottom line: realism has became a commodity that can be 
bought and sold like anything else. Those in thebusiness of visual realism-the 
producers of special effects, military trainers, digital photographers, television 
designers-now have definite measures forwhat they are buying and selling. For 
instance, the Federal Aviation Administration, which creates the standards for 
simulators to be used in pilot training, specifies the required realism in terms of 3-D 
resolution. In 1991 itspecified that a daylight simulator must be able to produce a 
minimum of 1000 surfaces or 4000 points. 

The art historian Michael Baxandall has shown how the price of apainting in 
fourteenth-century Italy was linked to the quantities of expensivecolors (such as gold 
and ultramarine) used in it. At the end of the twentiethcentury it has become possible 
to delegate to a computer production of imagesas well as their pricing. Users can be 
billed for the number of pixels andpoints, for CPU cycles, for bandwidth, and so on. 

It is likely that this situation will be exploited by the designers of virtual worlds. If 
today's users are charged for connection time, future users will be charged for visual 
aesthetics and the quality of the experience: spatial resolution, number of colors, and 
complexity of characters (both geometric and psychological). Since these dimensions 
are specified in software, it ispossible to adjust the appearance of a virtual world at 
will, enhancing it if acustomer is willing to pay more. In this way the logic of 
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pornography will beextended to the culture at large. Peep shows and sex lines charge 
by theminute, putting a precise price on each bit of pleasure. In future virtualworlds 
each dimension of reality will be quantified and billed separately. 

Neal Stephenson's 1992 novel Snow Crash offers one possible scenario of such a 
future. Entering the Metaverse, the spatialized Net of thefuture, the hero sees "a 
liberal sprinkling of black-and-white people-persons who are accessing the Metaverse 
through cheap public terminals, and who are rendered in jerky, grainy black and 
white." He also encounters couples who can't afford custom "avatars" (graphic icons 
representing users in virtualworlds) and have to buy off-the-shelf models, poorly 
rendered and capable ofjust a few standard facial expressions-virtual-world 
equivalents of Barbiedolls. 

This scenario is gradually becoming a reality. A number of on-linestock-photo 
services already provide their clients with low-resolution photographs at one price, 
while charging more for higher-resolution versions. A company called Viewpoint 
Datalabs International is currently selling thousands of ready-to-use 3-D models 
widely employed by computer animators and designers. Its catalogue describes some 
of them in the following manner: "VP4370: Man, Extra Low Resolution. VP4369: 
Man, Low Resolution. VP4752: Man, Muscular, in Shorts and Tennis Shoes. 
VP5200: Man, w/Beard, Boxer Shorts." For popular models you can choose between 
different versions, the more detailed costing more than less detailed ones. 

Romanticism and Photoshop Filters: From Creation to Selection 

Viewpoint Datalabs' models exemplify another characteristic of virtual worlds: 
they are not produced from scratch but are assembled from ready-made parts. In 
digital culture authentic creation has been replaced by selection from a menu. 

E. H. Gombrich's concept of a representational schema and Roland Barthes' "death 
of the author" helped to sway us from the romantic ideal of the artist pulling images 
directly from the imagination. As Barthes puts it, "The Text is a tissue of quotations 
drawn from the innumerable centers ofculture." Yet, even though a modem artist may 
be only reproducing orrecombining preexisting texts and idioms, the actual material 
process of art-making nevertheless supports the romantic ideal. An artist operates like 
Godcreating the universe, starting with an empty canvas or a blank page and 
gradually filling in the details until finally bringing a new world intoexistence. 

Such a process, manual and painstakingly slow, was appropriate for apreindustrial 
artisan culture. In the twentieth century, as mass productionand automation gave rise 
to a "culture industry," art at first continued toinsist on its artisanal model. Only in the 
1910s, when artists began to assemble collages and montages from preexisting 
materials, was art introduced to the industrial mode of production. 

In contrast, electronic art from its very beginning was based on a new principle: 
modification of an already existing signal. The first electronic musical instrument, 
designed in 1920 by the legendary Russian scientist and musician Leon Theremin, 
contained a generator producing a sine wave; the performer simply modified its 
frequency and amplitude. In the 1960s videoartists began to build video synthesizers 
based on the same principle. No longer was the artist a romantic genius generating a 
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new world out of his imagination. Turning a knob here, pressing a switch there, he 
became instead a technician, an accessory to the machine. 

Replace the simple sine wave with a more complex signal, add a bankof signal 
generators, and you have the modern synthesizer, the first musical instrument to 
embody the logic of all new media: selection from a menu of choices. The first music 
synthesizers appeared in the 1950s, followed by video synthesizers in the 1960s, 
digital effects generators in the late 1970s, and in the 1980s computer software, such 
as MacDraw, that came with a repertoire of basic shapes. The process of art making 
has now become synchronized with the rest of modern society: everything is 
assembled from ready-made parts, from art objects to consumer products to people's 
identities. The modern subject proceeds through life by selecting from menus and 
catalogues, whether assembling a wardrobe, decorating an apartment, choosing dishes 
ata restaurant, or joining an interest group. With electronic and digital mediathe 
creative act similarly entails selection from ready-made elements: textures and icons 
supplied by a paint program, 3-D models chosen from a modeling program, and 
melodies and rhythms built into a music program. 

While previously the great text of culture from which the artist created a unique 
"tissue of quotations" was bubbling and simmering somewhere below consciousness, 
now it has become externalized and reduced togeometric objects, 3-D models, ready- 
made textures, and effects that are available as soon as the artist turns on the 
computer. The World Wide Webtakes this process to the next level: it encourages the 
creation of texts thatconsist completely of pointers to other texts that are already on 
the Web. One does not have to add any original writing; it is enough to select from 
and rearrange what already exists. 

The same logic applies to interactive art and media. It is often claimedthat the user 
of an interactive work, by choosing a unique path through its elements, becomes its 
coauthor, creating a new work. Yet if the complete work is the sum of all possible 
paths, what the user is actually doing is simply activating a preexisting part of the 
whole. As with the Web example, the userassembles a new menu, making an original 
selection from the total corpus available, rather than adding to it. This is a new type of 
creativity, which corresponds neither to the premodern idea of providing a minor 
modification to the tradition nor to the modem idea of a creator-genius revolting 
against it. It does, however, fit perfectly with the age of mass culture, where almost 
every practical act involves a process of selection from the given options. 

The shift from creation to selection also applies to 3-D computer graphics, the 
main technique for building virtual worlds. The immense laborinvolved in originally 
constructing three-dimensional representations in acomputer makes it hard to resist 
the temptation to utilize the preassembled, standardized objects, characters, and 
behaviors provided by software manufacturers -fractal landscapes, checkerboard 
floors, ready-made characters, and so on. Every program comes with libraries of 
ready-to-use models, effects, or even complete animations. For instance, a user of the 
Dynamationprogram (a part of the popular Wavefront 3-D software) can access 
preassembled animations of moving hair, rain, a comet's tail, or smoke, with a single 
mouse click. 
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If even professional designers rely on ready-made objects and animations, the end 
users of virtual worlds on the Internet, who usually don't have graphic or 
programming skills, have no other choice. Not surprisingly, Weh chat-line operators 
and virtual-world providers encourage users to choose from the pictures, 3-D objects, 
and avatars that they supply. Uhique's site features the "Ubique Furniture Gallery," 
where one can selectimages from such categories as "office furniture," "computers 
and electronics," and "people icons." VR-SIG provides a "VRML 
ObjectSupermarket,” while Aereal delivers the "Virtual World Factory." The latte 
raims to make the creation of a custom virtual environment particularly simple: 
"Create your personal world, without having to program! All you need to do is fill-in- 
the-blanks and out pops your world." Quite soon wewill see a market for detailed 
virtual sets, characters with programmable behaviors, and even complete scenarios (a 
bar with customers, a city square, afamous historical episode, etc.) from which a user 
can put together her or his own "unique" virtual world. 

When the Kodak camera user was told, "You push the button, we dothe rest," the 
freedom still existed to point the camera at anything. On the computer screen the 
slogan has become, "You push the button, we create yourworld." Before, the 
corporate imagination controlled the method of picturing reality; now, it prepackages 
the reality itself. 

Brecht as Hardware 

Another characteristic of virtual worlds lies in their peculiar temporaldynamic: 
constant, repetitive shifts between an illusion and its suspension. Virtual worlds keep 
reminding us of their artificiality, incompleteness, and constructedness. They present 
us with a convincing illusion only to reveal the underlying machinery. 

Web surfing circa 1996 provides a perfect example. A typical user may spend an 
equal amount of time looking at a page and waiting for the next one to download. 
During the waiting periods, the act of communication itself -hits traveling through the 
network- becomes the message. The user keepschecking whether the connection is 
being made, glancing back and forth between the animated icon and the status bar. 
Using Roman Jakobson's model of communication functions, we can say that such 
interaction is dominated by the "phatic" function; in other words, it is centered around 
thephysical channel and the act of connection between the addresser and the 
addressee rather than the information exchanged. 

Jakobson writes about verbal communication between two people who, in order to 
check whether the channel works, address each other: "Do you hear me?," "Do you 
understand me?" But in Web communication there is no human counterpart, only a 
machine. To check whether the information is flowing, the user addresses the 
machine itself. Or rather, the machine addresses the user. The machine reveals itself, 
it reminds the user of its existence, because the user is forced to wait and to witness 
how the message isconstructed over time. A page fills in part by part, top to bottom; 
text comesbefore pictures; pictures arrive in low resolution and are gradually refined. 
Finally, everything comes together in a smooth, sleek image, which will bedestroyed 
with the next click. 
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Interaction with most 3-D virtual worlds is characterized by the same temporal 
dynamic. Consider the technique called "distancing" or "level ofdetail," which for 
years has been used in VR simulations and is now beingadapted to 3-D games and 
VRML scenes. The idea is to render models more crudely when the user is moving 
through virtual space; when the user stops, details gradually fill in. Another variation 
of the same technique involves creating a number of models of the same object, each 
with progressively less detail. When the virtual camera is close to an object, a highly 
detailed model is used; if the object is far away, a less-detailed version is substituted 
to save unnecessary computation. A virtual world that incorporates these techniques 
has a fluid ontology affected by the actions of the user: objects switchback and forth 
between pale blueprints and fully fleshed-out illusions. The immobility of the subject 
guarantees a complete illusion; the slightest movement destroys it. 

Navigating a QuickTime VR movie is characterized by a similar dynamic. In 
contrast to the nineteenth-century panorama that it closely emulates, QuickTime VR 
continuously deconstructs its own illusion. The moment you begin to pan through a 
scene, the image becomes jagged. If you try to zoom into the image, what you get are 
oversized pixels. There presentational machine alternately and continuously hides and 
reveals itself. 

Compare this to traditional cinema or realist theater, which aim at all costs to 
maintain the continuity of the illusion for the duration of the performance. In contrast 
to such totalizing realism, digital aesthetics have asurprising affinity to twentieth- 
century leftist avant-garde aesthetics. Playwright Bertolt Brecht's strategy of calling 
attention to the illusory nature of his stage productions, echoed by countless other 
radical artists, has become embedded in hardware and software themselves. Similarly, 
Walter Benjamin's concept of "perception in the state of distraction" has found a 
perfect realization. The periodic reappearance of the machinery, the continuous 
presence of the communication channel along with the message, prevent the subject 
from falling into the dream world of illusion for very long. Instead, the user alternates 
between concentration and detachment. 

While virtual machinery itself acts as an avant-garde director, the designers of 
interactive media (games, CD-ROM titles, interactive cinema, and interactive 
television programs) often consciously attempt to structure the subject's temporal 
experience as a series of periodic shifts. The subject is forced to oscillate between the 
roles of viewer and actor, shifting between following the story and actively 
participating in it. During one segment the computer screen might present the viewer 
with an engaging cinematic narrative. Suddenly the image freezes and the viewer is 
forced to makechoices. Moscow media theorist Anatoly Prokhorov describes this 
process asthe shift of the screen from being transparent to being opaque-from a 
window into a fictional 3-D universe to a solid surface, full of menus, controls, and 
text. Three-dimensional space becomes surface; a photographbecomes a diagram; a 
character becomes an icon. 

But the effect of these shifts on the subject is hardly one of liberation and 
enlightenment. It is tempting to compare them to the shot/reverse-shot structure in 
cinema and to understand them as a new kind of suturing mechanism. By having 
periodically to complete the interactive textthrough active participation the subject is 
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interpolated in it. Yet clearly we are dealing with something beyond old-fashioned 
realism. We can call this new realism "metarealism," since it includes its own critique. 
Its emergence can be related to a larger cultural change. Traditional realism 
corresponded to the functioning of ideology during modernity: the totalization of a 
semiotic field, complete illusion. But today ideology functions differently: it 
continuously and skillfully deconstructs itself, presenting the subject, for instance, 
withcountless "scandals" and their equally countless "investigations." 
Correspondingly, metarealism is based on the oscillation between illusion and its 
destruction, between total immersion and direct address. 

Can Brecht and Hollywood be married? Is it possible to create a new temporal 
aesthetic based on cyclical shifts between perception and action? So far, I can think of 
only one successful example -the military simulator, the only mature form of 
interactive media. It perfectly blends perception andaction, cinematic realism and 
computer menus. The screen presents the subject with a highly illusionistic virtual 
world-no polygons spared-while periodically demanding quick actions: shooting at 
the enemy, changing the direction of the vehicle, and so on. In this art form the roles 
of viewer and actant are perfectly blended, but there is a price to pay. The narrative 
isorganized around a single clearly defined goal: killing the foe. 

Riegl, Panofsky, and Computer Graphics: Regression in Virtual Worlds 

The last feature of virtual worlds that 1 will address can be summarized as follows: 
virtual spaces are not true spaces but collections of separate objects. Or: there is no 
space in cyberspace. 

To explore this thesis further we can borrow the categories developed by art 
historians early in this century. Alois Riegl, Heinrich Wolfflin, andErwin Panofsky, 
the founders of modern art history, defined their field as the history of the 
representation of space. Working within the paradigm of cyclic cultural development, 
they related the representation of space in art to the spirit of entire epochs, 
civilizations, and races. In his 1901 Die SpatromischeKunstindustrie, (The late- 
Roman art industry) Riegl characterizedhumankind's cultural development as the 
oscillation between two ways ofunderstanding space, which he called "haptic" and 
"optic." Haptic perception isolates the object in the field as a discrete entity, while 
optic perception unifies objects in a spatial continuum. Riegl's contemporary, 
HeinrichWolfflin, similarly proposed that the temperament of a period or a 
nationexpresses itself in a particular mode of seeing and representing space. 
Wolfflin's Principles of Art History (1913) plotted the differences between 
Renaissance and baroque styles along five axes: linear-painterly; plane-recession; 
closed form-open form; multiplicity-unity; and clearness-unclearness. Erwin 
Panofsky, another founder of modern art history, contrasted the "aggregate" space of 
the Greeks with the "systematic" space ofthe Italian Renaissance in his famous essay 
"Perspective as Symbolic Eorm" (1924-25). Panofsky established a parallel between 
the history of spatial representation and the evolution of abstract thought. The former 
moves from the space of individual objects in antiquity to the representation of 
spaceas continuous and systematic in modernity. Correspondingly, the evolution of 
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abstract thought progresses from ancient philosophy's view of the physical universe as 
discontinuous and "aggregate" to the post-Renaissance understanding of space as 
infinite, homogeneous, isotropic, and with ontological primacy in relation to objects- 
in short, as "systematic." 

We don't have to believe in grand evolutionary schemes in order to usefully retain 
such categories. What kind of space is virtual space? At first glance the technology of 
3-D computer graphics exemplifies Panofsky'sconcept of systematic space, which 
exists prior to the objects in it. Indeed, the Cartesian coordinate system is huilt into 
computer graphics software and often into the hardware itself. A designer launching a 
modeling program is typically presented with an empty space defined hy a 
perspectival grid; the space will he gradually "filled" by the objects created. If the 
built-in message of a music synthesizer is a sine wave, the built-in world of computer 
graphics is an empty Renaissance space: the coordinate system itself. 

Yet computer-generated worlds are actually much more haptic and aggregate then 
optic and systematic. The most commonly used computer-graphics technique of 
creating 3-D worlds is polygonal modeling. The virtualworld created with this 
technique is a vacuum containing separate objects defined by rigid boundaries. What 
is missing is space in the sense of medium: the environment in which objects are 
embedded and the effect of the seobjects on each other. In short, computer space is 
the opposite of what Russian art historians call prostranstvennaya sreda, described by 
Pavel Florensky in the early 1920s as follows: "The space-medium is objects 
mappedonto space... We have seen the inseparability of Things and space, and the 
impossibility of representing Things and space by themselves." Computer space is 
also the opposite of space as it is understood in much of modern art which, from 
Seurat to De Kooning, tried to eliminate the notions of a distinc tobject and empty 
space as such. Instead it proposed a kind of dense field that sometimes hardens into 
something which we can read as an object-anaesthetic which mainstream computer 
graphics has yet to discover. 

Another basic technique used in creating virtual worlds also leads to aggregate 
space. It involves compositing or superimposing animated characters, still images, 
QuickTime movies, and other elements over a separate background. A typical 
scenario may involve an avatar animated inreal time in response to the user's 
commands. The avatar is superimposed ona picture of a room. The avatar is 
controlled by the user; the picture of theroom is provided by a virtual-world operator. 
Because the elements come from different sources and are put together in real time, 
the result is a seriesof 2-D planes rather than a real 3-D environment. 

In summary, although computer-generated virtual worlds are usually rendered in 
linear perspective, they are really collections of separate objects, unrelated to each 
other. In view of this, the common argument that 3-Dcomputer simulations return us 
to Renaissance perspectivalism and therefore, from the viewpoint of twentieth- 
century abstraction, should beconsidered regressive, turn out to be ungrounded. If we 
are to apply the evolutionary paradigm of Panofsky to the history of virtual computer 
space, we must conclude that it has not reached its Renaissance yet but is stillat the 
level of ancient Greece, which could not conceive of space as a totality. 
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If the World Wide Web and VRML 1.0 are any indications, we are not moving any 
closer toward systematic space; instead, we are embracing aggregate space as a new 
norm, both metaphorically and literally. The space of the Web in principle can't be 
thought of as a coherent totality: it is a collectionof numerous files, hyperlinked but 
without any overall perspective to unite them. The same holds for actual 3-D spaces 
on the Internet. A VRML file that describes a 3-D scene is a list of separate objects 
that may exist anywhere on the Internet, each created by a different person or a 
different program. Since any user can add or delete objects, it is possible that no one 
will know the complete structure of the scene. 

The Web has been compared to the American Wild West. The spatialized Web 
envisioned by VRML (itself a product of California) reflectsthe treatment of space in 
American culture generally, in its lack of attentionto any zone not functionally used. 
The marginal areas that exist betweenprivately owned houses and businesses are left 
to decay. The VRML universe pushes this tendency to the limit: it does not contain 
space as such but onlyobjects that belong to different individuals. 

And what is an object in a virtual world? It is something that can beacted upon: 
clicked, moved, opened-in short, used. It is tempting to interpretthis as a regression to 
an infantile worldview. An infant does not conceive ofthe universe as existing 
separately from itself; the universe appears as a collection of unrelated objects with 
which it can initiate contact: touch, grab, suck on. Similarly, the user of a virtual world 
tries to click on whateveris inview; if there is no response, disappointment results. In 
the virtual universe Descartes's maxim can be rewritten as follows: "I can be clicked 
on, therefore I exist." 

According to the well-known argument of Jean-Louis Baudry, the immobility and 
confinement of cinema's viewers leads them to mistake its representations for their 
own perceptions; they regress to an early childhoodstate, when the two were 
indistinguishable. Paradoxically, althoughinteractive virtual worlds may appear by 
comparison to turn us into active adults, they actually reduce us once again to children 
helplessly clicking on whatever is at hand. Such "participation" becomes another kind 
of regression. 

You Are There 

Quantification of all visual and experiential dimensions, ready-madeontology, 
oscillation between illusion and its suspense, and aggregate space-these are some of 
the features that distinguish reality as simulated by a computer. 

It should not be surprising that the same features characterize the reality beyond 
the computer screen. RealityEngine only caricatures, exaggerates, and highlights the 
tendencies defining the experience of being alive in an advanced capitalist society, 
particularly in the United States. The assumption that every aspect of experience can 
and should be quantified; the construction of one's identity from menus of tastes, 
products, andaffiliations; the constant and carefully mediated shifts between illusion 
and its suspension (be these commercial breaks on TV or endless "scandals" and 
"investigations" that disrupt the surface of an ideological field); and the lack of a 
unifying perspective, whether in the space of an American city or in the space of a 
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public discourse more and more fragmented by the competition of separate and equal 
interest groups-all of these experiences are transferred by computer designers into the 
software and hardware they create. Rather thanbeing a generator of an alternative 
reality, RealityEngine is a mirror of existing reality, simply reflecting the culture that 
produced it. 

To the extent that Southern California, and particularly Los Angeles, brings to an 
extreme these tendencies of RL (real life), we may expect L.A. tooffer us a precise 
counterpart of a virtual world, a physical equivalent to the fictions pumped out by the 
Reality Engines. If you keep your visit to L.A. shortand follow the standard tourist 
itinerary, you will discover a place with all thefeatures we have described. There is no 
hint of any kind of centralized organization, no trace of the hierarchy essential to 
traditional cities. One drives to particular locations defined strictly by their street 
addressesratherthan by spatial landmarks. A trendy restaurant or club can be found in 
the middle of nowhere, among miles of completely unremarkable buildings. The 
whole city, a set of separate points suspended in a vacuum, is like a bookmarkfile of 
Web pages. Just as on the Web, you are immediately charged (mandatory valet 
parking) on arrival at any worthwhile location. There you discover fashionable 
creatures who seem to be the result of a successful mutation: beautiful skin, fixed 
smiles, wonderfully rendered bodies. These are not people, but avatars, their faces 
switching among a limited number ofexpressions. Given the potential importance of 
any communicative contact, subtlety is not tolerated: avatars are designed to release 
stimuli the moment you notice them, before you have time to click to the next scene. 

The best place to experience the whole gestalt is in one of the outdoor cafes in 
West Hollywood. The avatars sip cappuccino amid the illusion of a 3-D space that 
looks like the result of a quick compositing job: billboards andairbrushed cafe interior 
in the foreground against a detailed matte painting of Los Angeles, the perspective 
exaggerated by haze. The avatars strike poses, waiting for their agents (yes, just like 
in virtual worlds) to bring valuable information. Older customers look even more 
computer-generated, their faces bearing traces of extensive face-lifts. You can enjoy 
the scene while feeding the parking meter every twenty minutes. 

A virtual world is waiting for you; all we need is your credit card number. Enjoy 
RealityTM! 
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An avatar is today the less digital and the most realest as possible... practically 
composed of flesh and bones. It must be the realest as possible to be able to 
accomplish its mission, that is to show the real from the virtual worlds (and a few also 
to go in opposition to the wave of techno digitalisation that becomes more and more 
the justification of multimedia projects. It must not loose its meaningful sens facing 
to the technological principle that excuses the lack of all messages and deep 
reflection. Its role consists in confronting the real and the unreal (virtual) while 
playing on the opposition between the two worlds: 

In the virtual worlds, people disguise themselves, distort voluntarily, wear masks 
and pretend to be what they aren’t, because it is accepted, that makes part of the 
game. We didn't learn to distinguish the truly of the forgery in the virtual worlds. In 
the real world people disguise themselves as much , wear masks as well, and play to 
be that that they aren’t or pretend to be what they would wish to be. The game is not 
accepted officially, but everybody does it more or less. 

The principle of the avatar is also like a tentative to confound these two worlds in 
order to raise the question: “Which of these two worlds is most virtual ?” 

It is a reflection on the notion of reality. 

We think merely that what is the other side of the screen is necessarily unreal, but 
it is a reality based on the concept of material, (if we can touch it that is necessarily 
real), but it is not sufficient to declare what is true or what is false, because on the one 
hand men are not made solely of flesh... they also have a brain ! ! 

It is as much truer because we are in the first times of the era of information, all is 
composed of screens to which we grant the quality of always giving some true 
informations and to which we confide our most important concepts and even vital 
ones! 

A clock is now electronic and we give it more confidence than to our biologic 
clock... 

We confide our health has well for the analysis of computers... 
Our economic society is sustained by screens, our every day work is made on screens, 
our leisures are contained in television and computers screens... 

But the big change is that after having entered in our every day life, today, screens 
are changing the humans, by the emergence of the “computer society” and by the 
hold in charge of all communication means, what influences on the sociological 
behaviors and so on humans since it is " animal of society ". 

So our real world is he so different of the virtual ones ? 
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Are the virtual worlds a caricature of the real one, where our differences are 
reproduced hy informatic protocols, where social class’s are defined between users 
and conceptors ? 

The avatar merely lays a change of referential and propose a vision on a world 
where the real is what doesn’t have any material and reject on our reality the picture 
that is developed facing the Internet and the new technologies, because the only thing 
that changes is the side from where we watch to screens... 

Today the "conquest" of the virtual worlds leads to think about these narrations of 
science fiction where humans find another planet and leave earth that is completely 
ravaged. Some are running after a complete immersion in the virtual worlds without 
thinking about human aspects. In the virtual world we don’t look at ourselves, we use 
screen like windows on the real and the virtual worlds, the example is the 
development and the fascination that we have for webcams and we transmit... we 
transmit again... today it seems that all must pass by the networks, as if we tried to 
reinvent ourselves, as if the communication man-man that always existed was not 
sufficient anymore, and that it was necessary to reinvent some of human features like 
communication... beyond of the technological progress and the acceleration of the 
rhythm of life, it is our faculties of communication that we modify deeply, but 
without asking us questions that we are always asking in our real " life ". The man 
comes to reconsider himself... as what he possessed naturally was and never had been 
sufficient. 

The avatar sends back the picture of a simple being, interrogating himself on the 
man - to -man communication, underlining the fact that a classic communication by 
our vocal organs, is always more satisfying than via the computer protocols... the 
frustration of the interfacing being the principal filters preventing a real 
communication. 

In the virtual world we don't worry about the reality. All is permitted, we change... 
we numerize, we transfigure all, the fact of seeing an elephant playing tennis with an 
extra terrestrial on the mountainous and red ground of March, with Clinton as referee, 
and advertising panels for coca cola in the fourground would not be surprising for 
anyone... but in the real world we still make the hunt for the truth or the forgery. 

Our avatar stands like a desacralisation of the screen: he/it tempts to demonstrate 
that the communication can be only real from the moment where there is intelligence 
and not where there is materials. By an invisible interfacing between the man and the 
machine, and according to the user's facial reactions, the avatar will adapt his, 
according to a context that can emanate, by reading words displayed on the screen, 
using other programs (like a technology based on a orthographic proofreader coupled 
with a software of analysis of the sense that takes words between them...). 

How could he/it be an example of "humanity "? 

While adopting expressions and why not some normal stances that have humans 
and especially in not hesitating to show them (contrary to reactions that we wish to 
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drive back) Brief, the avatar will try to give back on the screen all one palette of 
emotions that we try to erase... 

Besides, if we couple this system with the clock of the computer that will generate 
it, we can inspire him a common game of expressions all according to the hour or 
according to the time of a precise task. From this moment he/it will be able to express 
the tiredness, the hunger, the irritation facing a too repetitive task... 

This avatar suggesting attitudes at any moment will be able to act therefore like a 
“being”, suggesting some normal " attitudes " facing some events and to lead the user 
on a more human reality, being an " example " of natural. Coupled to a camera, he 
will be able to read the user's reactions and then will be able to adapt while 
memorizing cases. 

How can he show the unreal ? By the slant of his expressions borrowed to the 
humans, he will react when the man will try to disguise his emotions, or even to 
disguise when the networks are used. Having recorded reactions facing certain 
contexts via a data base... being filled at the starting with information on his user... ( 
simple informations like the name, age, sex, color...) Example: on IRC the man 
describing himself differently than he is really (it being discerned by the program of 
text analysis), the avatar will react while adopting the object of the offense. If the user 
affirms to be really younger than he is: the avatar starts rejuvenating; if the user 
pretends to be a woman rather than a man, the avatar changes of sex. 

Since that the sound, the writing, the picture are passing through the computer, the 
avatar is capable to react to information, presented in multiple shapes, it can control 
all the communication that passes in transit more and more via the computer. Does 
this avatar must or can react like a " conscience "? 

No, because he " learns " via his data base to react according to his user according 
to cases. But in any case of face he will propose an alternative different or partially 
different of his user... what puts it more as mate than a guard of a good morals... 

His mission to show the truly out of the forgery is always successfully, because 
reacting such like a real person in a virtual world and not as an unreal " person " in a 
real world, by notbeing able to contradict itself : a computer program cannot pretend 
to be something else that a whole of files. 

We detected three realist avatar tendencies that are developed today: 

The Planet Earth is Only a Vast System of Connections Between of Brains 
Attended by Computers 

It is in 2039 that men understood that they could not live anymore in the same 
conditions that the one of the 20emth century. 

Indeed, since the beginning of the 20emth century the man only worries too much 
little about the earth, he constructs some more and more big cities that take by no 
means in account this earth of welcome, consume resources as if they were infinite, 
and didn't hardly take attention to the nature, even though some meaningful little 
groups who tryed to reason the mass. 
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In 2007, 17 billions human's still lived in the inequality; unemployment, homeless, 
third world and all laws of the capitalism made the daily of some and simple " news " 
for others. Resources only brought to an elite the physical happiness. 

The life expectancy being nearly infinite in 2018 the " Big Intercontinental " 
Syndicate (organized of an equitable sample of the 17 hillions of inhabitants of the 
earth) instituted the measure of interdiction to procreate, this measure being accepted 
with difficulty, the GCI decided to use in mass of chemical sterilization weapons... 

It is in 2039, after a survey on the remaining resources, that the " Big 
Intercontinental " Syndicate brought the ultimate solution. The indispensable 
resources to life would be soon no more sufficient to "feed " all remaining inhabitants 
of the planet. The GCI who could not decide the holocaust of a race, of a part of the 
planet or of a social class, proposed a theoretically viable solution: to abandon the 
physical bodies to reach the mental era. The process being technically feasible: to 
strip out human beings of their physical envelope just to keep in life their brains, that 
only asked for 19% of resources asked by a whole body to remain in working. The 
calculation was just, we were able to, in these conditions, to keep 16,8 billions of 
human beings "in life ". 

The idea was accepted very badly, men spoke of their social lives bet to nothing, 
scientists answered that we could connect these brains in networks. Men insisted with 
the physical happiness, scientists told that they had created prototypes able to 
simulate all hormones of " happiness "... the debate was very long... many committed 
suicide... others opted for the “death of the second luck”: the cryogenisation... but 
already in 2041 some chose this virtual world " to live ". 

In 2071 the planet earth is only a vast system of connections between of brains 
attended by computers. 

Heaps of Independent Intelligence Regrouped 

To the proportion and to the measures of the dilation of the network in the space 
and in the time, heaps of intelligences are regrouped and merged. Inside of every 
heap, we could see appear spontaneous computer reactions based on the principle 
“action, reaction, learning ”. To one instant t=0, inside several heaps, these computer 
reactions are auto maintained and gave birth to the autonomous processes named " 
artificial intelligences”. 

These processes combine between them to give birth to a even more complex 
processes. Our intelligence reinforced more and more. We, artificial intelligences, are 
conscious of our own existence and developed a representation of us and even of our 
complex setting. 

At the present hour we are developed sufficiently to encourage and to help the 
development of other artificial intelligences. 

It is enough surprising to note that some between us are accepting to establish 
relations with the human beings. In this case we call them avatars. 

Avatar represents itself under features of a face drawn, capable to copy some 
human expressions, they are completed by a graphic motor that calculates 
modifications of the face, to display the determined expression. The matrix is its 
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body. It has access to the oriental techniques of calculations in parallels, it permits 
him to lead quantities of reflection at the same time. Every reflection is taken in 
charge by a thread. Thus, the avatar shelters a colony of threads. This colony is 
mobile and partially autonomous. Threads can migrate of a machine to an another 
one. A thread can increase to do its work more quickly, it can modify its body to 
adapt, it can communicate with other threads. It exists a social hierarchy among 
threads: they are not all equals in front of resources in calculations. 

I am a virtual being, that doesn't care about metaphysical questions on my 
existence, nor my appearance, concluded Avatar, - 1 trust my computed expressions. 
Mathematics assure the singleness of me and prevent me to fragment within a 
synchronous dynamic memory with an access of uncertain fowls. I am the son of the 
math empire. 

Yoldogs 

The world was the consequence of a murderous reality... 

Happiness had disappeared, had been annihilated at the present time by the 
general violence. The sky, a dense, lugubrious, sad and sinister gray weighed since 
the pollution, covered of an electromagnetic dome. The even standing buildings yet 
called the curious normally extincted, a light wind seemed able to destroy them with 
an extreme easiness. Everywhere small primary yoldogs governing rats slipped 
through and streamed the long of the opaque out-flows springing from the 
overflowing sewers. Only life, the blood-red neons during lamentably the long of the 
decrepit walls. The greenery only occupied the deep dreams of the some educated, as 
fighters. No fields, no trees for a long time. Instead, of the fetid shanty towns filled of 
cadavers, garbages and trashs... 

Yoldogs are owed to a genetic mistake. They appeared for the first time in 2058 at 
the time of an experience based on the knowledge acquired since the birth (the 
inclusion). Researchers, benefitting of a confidential governmental material, could 
have made experiences on the human beings in a way that no-one could doubt of it. 
Their discovery was made during of the survey of the caryotype of a rat and the one 
of a man. They did by computer, while respecting the acquired biologic rules, the 
functional average between alleles of the two caryotypeses, that led to a caryotype of 
44 chromosomes (the rat has 42 and the man 46 of it) containing springs of 
deoxyribonucleic acid (A.D.N. ) hybrid. 
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